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Preface 


This  handbook  provides  guidance  in  evaluating  community-based  partnerships 
whose  goals  include  prevention  and  control  of  cardiovascular  diseases  (CVD). 
Such  coalitions  and  projects  work  with  the  community  to  reduce  the  prevalence 
of  risk  factors  for  CVD. 


Priority  areas  often  include: 

•  Improving  nutrition  and  diet. 

•  Increasing  physical  activity. 

•  Preventing  or  reducing  tobacco  use. 


Community-based  efforts  to  promote  health  and  prevent  disease  go  by  a  variety 
of  names,  including  coalitions,  initiatives,  and  partnerships.  We  use  these 
several  related  terms  interchangeably.  Each  conveys  the  idea  of  collaboration: 
building  a  new  pattern  of  relationships  among  multiple  participants  for  a  shared 
purpose,  such  as  reducing  risks  for  CVD.  Coalitions  enhance  the  opportunities 
for  citizens  to  develop  healthy  diets  and  exercise  habits  and  to  stop  or  avoid 
tobacco  use. 

Community  partnerships  involve  many  parts  of  the  community,  such  as  schools, 
worksites,  and  religious  organizations,  to  make  changes  that  will  reduce  risks  for 
CVD  both  at  the  individual  and  population  levels.  For  example,  a  community 
coalition  for  heart  health  might  encourage  schools  to  modify  lunch  menus, 
religious  organizations  to  establish  walking  programs,  and  worksites  to  establish 
smoke-free  environments.   Coalitions  reduce  risk  by  changing  programs, 
policies,  and  practices  in  all  relevant  sectors  of  the  community. 

The  strength  of  community  partnerships — facilitating  a  variety  of  changes  in 
multiple  sectors — is  also  what  makes  them  difficult  to  evaluate.  Traditionally, 
evaluation  methods  were  geared  to  understanding  whether  a  particular  program 
had  an  effect  on  a  particular  outcome.  It  is  more  difficult  to  establish  cause  and 
effect  with  community  partnerships  where  a  variety  of  citizens  work  to  produce 
changes  in  many  parts  of  the  community  at  the  same  time. 

If  evaluation  of  community  partnerships  is  so  difficult,  why  bother? 
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Consider  whether  any  of  the  following  are 
compelling  reasons  to  evaluate 

Help  partnerships  review  their  progress  and  continually  improve  their 
efforts. 

Gain  community  support,  grant  money,  and  donations. 

Help  overcome  resistance  to  the  initiative. 

Determine  whether  the  partnership  has  attained  its  goals. 

Detect  successful  projects  early  in  the  life  of  the  coalition. 

Identify  unforeseen  challenges  and  side  effects  and  redirect  efforts  of  the 
coalition. 

Acknowledge  and  celebrate  small  wins  on  the  road  to  successful  outcomes. 

Be  more  responsive  and  accountable  to  community  members, 
grantmakers,  and  other  constituents. 


This  handbook  offers  some  practical  tools  that  can  be  used  to  meet  these  and 
other  purposes  of  evaluation.   Using  some  or  all  of  these  tools  can  help  us 
understand  and  improve  community  partnerships  for  promoting  nutrition, 
physical  activity,  tobacco  control,  and  other  heart  health  issues. 

The  purpose  of  this  evaluation  handbook  is  to  provide  tools  and  strategies  for 
measuring  the  products,  effects,  and  outcomes  resulting  from  the  efforts  of  a 
community  initiative  to  prevent  CVD.   Other  coalition  assessment  guides  have 
often  focused  more  on  the  process  of  coalition  building,  such  as  the  number  or 
type  of  members  or  meetings  held.  Because  an  emphasis  in  this  handbook  is  on 
monitoring  community  action  and  change,  this  system  helps  connect  the  actions 
of  partners  to  the  effects  upon  a  community.  When  combined  with  other 
measures  of  process  and  outcome,  such  as  member  satisfaction  and  behavioral 
surveys  and  community-level  indicators  of  change,  this  system  provides  a 
comprehensive  evaluation  of  CVD  prevention  efforts. 

This  evaluation  system  is  flexible:  it  may  be  used  in  its  entirety  or  in  its  various 
parts.   It  may  be  used  by  people  internal  or  external  to  the  initiative.  The 
handbook  offers  a  number  of  tools  that  staff  and  evaluators  of  community 
partnerships  can  use  to  help  understand  and  improve  efforts  to  reduce  risks  for 
CVD. 
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The  specific  components  and  tools  chosen  to  evaluate  a  given  community 
partnership  will  be  unique  to  each  situation  and  to  the  level  of  resources 
available  for  evaluation.  A  distinction  between  groups  with  "High"  and  "Low" 
resources  and  requirements  for  the  evaluation  may  be  helpful. 

The  HIGH  groups  are  those  with  abundant  resources  and  extensive 
evaluation  requirements.  Users  from  the  HIGH  groups  would  need  an 
adequate  budget  for  staffing  and  paying  for  expenses  related  to  the  evaluation. 
HIGH  users  would  need  staff  to  collect  and  provide  information  plus  skilled 
evaluators;  funders  or  grantmakers  would  likely  expect  that  a  variety  of 
questions  would  be  addressed  by  a  comprehensive  evaluation.  HIGH  users 
should  consider  addressing  all  of  the  components  in  the  evaluation  system  and 
answering  all  of  the  evaluation  questions  outlined  in  Chapters  4-9. 

The  LOW  groups  have  fewer  resources  and  few  or  no  evaluation 
requirements.  Users  from  the  LOW  groups  may  have  limited  staff  time  for 
supporting  coalition  activities  and  virtually  no  resources  for  evaluation.  If  there 
is  any  grant  support,  there  may  be  little  or  no  expectations  for  reporting  and 
evaluation.  Staff  might  be  available  for  evaluation  activities  only  a  few  hours  per 
week  or  less  and  have  no  or  minimal  training  in  evaluation.  LOW  users  should 
consider  prioritizing  evaluation  questions  and  components,  addressing  a  subset 
of  questions  of  interest. 

At  various  points  in  this  handbook,  we  provide  suggestions  to  guide  groups  with 
LOW  and  HIGH  resources  and  evaluation  requirements  in  tailoring  the  system 
to  meet  their  needs.  Additionally,  each  chapter  of  the  evaluation  system 
(Chapters  4-9)  has  a  section  of  recommended  strategies  for  those  operating  on 
a  shoestring  budget  for  evaluation.  By  applying  the  suggestions  for  tailoring  the 
evaluation  system,  community-level  practitioners  will  be  able  to  complete  useful 
and  valid  evaluations  of  their  programs  even  though  resources  may  be  limited. 

This  handbook  is  divided  into  three  parts.  Part  I  introduces  the  user  to  CVD  and 
community-based  prevention  and  evaluation,  key  evaluation  questions  for  CVD 
prevention  initiatives,  and  the  framework  for  monitoring  and  improving  CVD 
programs.   Part  II  presents  each  component  of  the  evaluation  system,  its 
application,  and  sample  tools.   Part  III  addresses  how  to  integrate  the 
information  to  address  key  questions  and  communicate  information  about  a 
CVD  prevention  initiative. 

The  evaluation  system  outlined  here  is  based  on  experiences  of  the  Work  Group 
on  Health  Promotion  and  Community  Development  at  the  University  of 
Kansas.1  These  experiences  involve  collaborations  with  nearly  20  different 
community  partnerships,  including  those  for  preventing  substance  abuse, 
adolescent  pregnancy,  and  risks  for  CVD.  This  monitoring  system2  has  also 
been  used  with  health  and  human  service  coalitions,  rural  health  promotion,  and 
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a  tribal  coalition  to  reduce  substance  abuse  among  Native  Americans.  This 
guide  for  evaluating  coalitions  for  prevention  of  CVD  is  a  specific  adaptation  of 
a  more  generic  handbook  for  evaluating  community  health  initiatives.3  Many  of 
the  examples  in  this  CVD  handbook  are  drawn  from  the  evaluation  of  Kansas 
Low-Fat  Eating  for  America  Now  (LEAN),  a  statewide  coalition  to  reduce  risks 
for  CVD  through  changes  in  diet  and  physical  activity.4 

This  handbook  provides  guidelines  and  tools  by  which  community  leadership, 
evaluators,  and  grantmakers  can  pose  and  address  questions  of  interest.  This  is 
not  a  self-contained  text  on  evaluation.  More  technical  sections  presume  a  level 
of  expertise  that  could  be  found  in  many  State  and  local  health  departments  and 
colleges  and  universities.   Our  hope  is  that  these  tools  can  promote 
accountability  and  improvement,  thus  extending  the  capacity  of  community 
partnerships  to  improve  health. 
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1 


Overview  of  Cardiovascular  Diseases 
and  Community-Based 
Prevention  Efforts 


CVD:  What  Is  It? 

CVD  is  the  leading  cause  of  death  and  disability  in  the  United  States.  CVD, 
including  coronary  heart  disease  and  stroke,  accounts  for  over  43%  of  all  deaths 
in  the  United  States,  and  it  is  the  major  cause  of  hospitalization  and  disability. 
CVD  refers  to  a  wide  variety  of  heart  and  blood  vessel  diseases,  including 
coronary  heart  disease,  hypertension,  stroke,  and  rheumatic  heart  disease. 
More  than  916,000  Americans  died  from  CVD  in  1990. 

The  two  major  forms  of  CVD  are  coronary  heart  disease  and  stroke.  Coronary 
heart  disease,  also  called  ischemic  heart  disease  or  coronary  artery  disease,  is  a 
term  used  to  identify  several  disorders  that  reduce  the  blood  supply  to  the  heart 
muscle.  The  most  common  types  of  CVD  are  angina  pectoris  (chest  pain), 
myocardial  infarction  (heart  attack),  and  sudden  death.  Stroke  or  cerebro- 
vascular disease  is  the  third  leading  cause  of  death  and  represents  about  one- 
sixth  of  all  CVD.  Stroke  includes  a  group  of  diseases  that  affect  the  arteries  of 
the  central  nervous  system.  Stroke  results  when  an  artery  in  the  brain  is  either 
ruptured  or  clogged  by  a  blood  clot  (thrombosis),  a  wandering  clot  (embolus),  or 
atherosclerotic  plaque. 


CVD:  What  Causes  It? 

Atherosclerosis  is  the  underlying  disease  process  of  the  major  forms  of  CVD.  It 
is  a  slowly  progressive  condition  in  which  the  inner  layers  of  the  artery  wall 
become  thick  and  irregular  because  of  plaque — deposits  of  fat  cholesterol  and 
other  substances.  CVD  is  usually  revealed  clinically  in  middle  age  or  later,  but 
atherosclerosis  begins  in  childhood.  Atherosclerosis  is  associated  with  several 
modifiable  risk  factors.  These  include  high  blood  pressure,  high  blood 
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cholesterol,  cigarette  smoking,  physical  inactivity,  obesity,  and  diabetes.  Control 
of  these  risk  factors  at  both  the  population  and  individual  levels  is  the  key  to 
prevention  of  CVD.    Although  most  of  the  major  risk  factors  for  CVD  are 
modifiable  or  entirely  preventable,  over  80%  of  Americans  report  at  least  one 
major  risk  factor. 


CVD:  How  Do  You  Prevent  It? 

Two  major  primary  prevention  strategies  have  been  used  in  public  health  to 
reduce  CVD.  The  high-risk  approach  aims  to  identify  high-risk  individuals 
through  population  screening  and  to  refer  them  for  treatment.  The  community- 
based  approach  aims  to  reduce  CVD  by  changing  behaviors,  policies,  and  the 
environments  of  whole  populations.  Although  the  high-risk  approach  has  its 
advantages,  the  pervasive  and  multifactorial  nature  of  CVD  makes  community- 
based  interventions  that  use  multiple  strategies  a  first  choice.  These  approaches 
are  comprehensive:  they  target  multiple  sectors  of  a  community,  such  as 
schools  and  health  organizations;  use  multiple  strategies,  such  as  information 
and  policy  change;  and  address  several  CVD  risk  factors,  such  as  diet  and 
physical  inactivity. 

One  of  the  oldest  community-based  CVD  programs  is  the  North  Karelia 
(Finland)  Project,  which  began  in  1972  and  provided  a  broad  range  of  risk  factor 
interventions,  such  as  messages  in  the  media  and  programs  for  improving  diet, 
physical  activity,  and  reducing  tobacco  use.  The  project  targeted  individuals  and 
groups  as  well  as  policies  and  environmental  conditions  that  discouraged 
healthy  lifestyles.  Recent  evidence  suggests  that  this  program  has  had  significant 
effects,  resulting  in  a  60%  decline  in  the  number  of  CVD  deaths. 

Early,  or  first  generation,  research  and  demonstration  programs  in  the  United 
States  were  supported  by  the  National  Heart,  Lung,  and  Blood  Institute.  These 
included  the  Stanford  Three  City  and  Five  City  Projects,  the  Minnesota  Heart 
Health  Program,  and  the  Pawtucket  Heart  Health  Program.  Each  of  these 
efforts  was  able  to  demonstrate  successful  interventions  for  reducing  risk  for 
some  of  the  population  or  targeted  risk  factors. 

Second  generation  programs,  such  as  the  New  York  Healthy-Heart  program 
and  the  South  Carolina  CVD  Prevention  Project,  funded  by  CDC,  demonstrated 
that  State  health  agencies  could  work  successfully  with  local  communities  to 
plan  and  conduct  community-wide  CVD  prevention  efforts.  Another  example, 
the  Missouri  Bootheel  Heart  Health  project,  was  a  State  Health  Agency-CDC 
collaboration  for  prevention  of  CVD  in  a  poor,  rural  six  county  area  in  the 
southeast  corner  of  the  state.  Successes  of  the  Bootheel  coalition  included 
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forming  walking  clubs,  building  walking  trails,  beginning  exercise  classes  in 
community  churches,  providing  blood  pressure  and  cholesterol  screening,  and 
mobilizing  the  community  around  preventing  heart  disease. 

These  community-based  partnerships  for  preventing  CVD  involve  community 
members  in  planning  and  implementing  local  efforts.  Perhaps  the  best  known 
variation  of  this  community  development  approach  is  the  Planned  Approach  to 
Community  Health  (PATCH).  PATCH  was  developed  in  the  mid-1980s  by  the 
CDC  in  partnership  with  State  and  local  health  departments  and  community 
groups.  It  is  a  participatory  model  of  community  development  that  includes 
active  involvement  of  coalition  members  in  planning  comprehensive  strategies 
for  health  promotion.  PATCH  programs  are  now  in  43  states,  the  District  of 
Columbia,  and  the  Virgin  Islands. 


Forming  Community  Partnerships 

A  variety  of  community  partnerships  have  formed  to  reduce  risks  for  CVD. 
These  partnerships  serve  as  catalysts  for  change:   bringing  about  new  or 
modified  programs,  policies,  and  practices  that  reduce  risks.  Using  methods  of 
community  planning,  these  initiatives  promote  a  variety  of  changes  across 
multiple  sectors  of  the  community  such  as  schools,  worksites,  and  businesses 
(see  Figure  1.1).  Healthy  People  2000  is  an  excellent  resource  for  new 
cardiovascular  health  initiatives.  The  health  promotion  and  disease  prevention 
objectives  that  relate  to  CVD  provide  guidance  in  establishing  appropriate 
public  health  goals.1 


Why  Another  Evaluation  Manual? 

Experiences  from  several  generations  of  community  programs  and  partnerships 
emphasize  challenges  in  monitoring  and  evaluating  program  effects  and  levels 
of  effort.  Program  effects  are  difficult  to  document  and  to  disentangle  from 
effects  of  other  efforts,  influences,  and  trends.  Data  from  experiments  with  well- 
controlled  interventions  for  targeted  risk  factors  are  easier  to  interpret.  We  can 
examine  the  effects  of  particular  interventions  on  particular  behaviors;  for 
example,  smoking  cessation  programs  to  get  people  to  quit,  physical  activity 
programs  to  motivate  people  to  exercise,  and  menu  marking  programs  to  affect 
restaurant  patrons'  choice  of  meals. 
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Figure  1.1 
Multiple  sectors  in  which  community  change  is  sought  to  prevent  CVD 


Documenting  such  effects  in  community-based  coalitions  and  partnerships  is 
more  difficult,  however,  since  the  goals  are  much  broader  and  more  diffuse.  (In 
fact,  in  some  cases  we  have  recommended  that  community-based  programs 
limit  themselves  to  process  evaluation,  leaving  evaluation  of  outcome  and 
impact  to  research  programs.  In  many  cases,  however,  both  process  and 
outcome  evaluations  are  required  by  program  sponsors.)  Process  measures, 
such  as  the  number  of  individuals  served  by  a  program,  or  changes  in  individual 
participant  behaviors,  such  as  reported  diet,  may  be  relatively  easy  to  collect.  By 
contrast,  changes  in  policy  or  systems,  changes  in  the  environment  facilitated  by 
community  partnerships,  such  as  the  addition  of  a  walking  trail  or  improved 
dietary  choices  in  cafeterias,  are  often  harder  to  document. 
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Community  partnerships  involve  citizens  in  setting  priorities  and  creating  new 
relationships  that  facilitate  community  change.  However,  the  lack  of  a 
monitoring  system  limits  opportunities  to  enhance  effectiveness  of  local  change 
efforts.  A  system  for  monitoring  early  successes  can  provide  community 
members,  program  leaders,  evaluators,  and  funders  with  the  opportunity  to 
celebrate  early  success  and  redirect  the  partnership's  energy  to  more  effective 
interventions. 


The  Handbook:  An  Evaluation  System 

The  evaluation  system  described  in  this  handbook  outlines  a  comprehensive 
evaluation  system  to  (a)  monitor  implementation  of  change  efforts  (such  as 
community  actions  to  mobilize  the  community),  (b)  assess  intermediate 
outcomes  (such  as  changes  in  programs,  policies,  and  practices  facilitated  by  the 
initiative),  and  (c)  collect  behavioral  and  community-level  measures  of  reduced 
risks  for  CVD  (such  as  reported  levels  of  physical  activity). 

Figures  1.2-1.4  illustrate  the  types  of  community  changes  that  a  partnership  for 
prevention  of  CVD  might  seek  in  different  sectors  of  the  community.2  Note  that 
a  variety  of  small  changes  in  programs,  policies,  and  practices  could  help  reduce 
risks  associated  with  diet,  tobacco  use,  and  physical  inactivity. 
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Figure  1.2 
Illustrative  community  changes  in  schools  to  improve  diet 
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Figure  1.3 

Illustrative  community  changes  in  the  business  community 

to  reduce  tobacco  use 
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Figure  1.4 

Illustrative  community  changes  in  the  broader  community 

to  increase  physical  activity 
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Evaluating  Community  Partnerships  to 
Prevent  Cardiovascular  Diseases: 
Some  Key  Questions 


Often,  community  partnerships  collect  data  that  are  analyzed,  filed,  and  never 
used.  Why?  Evaluators  may  have  chosen  poor  measures  of  community 
efforts — measures  that  do  not  really  capture  what  made  the  initiative  unique  or 
effective.  Perhaps  the  measures  were  chosen  well  but  evaluation  data  were 
never  provided  to  project  staff  in  a  useful  and  understandable  format.  When  we 
are  involved  in  efforts  to  improve  the  health  of  our  communities,  we  want  to 
know  whether  we  are  achieving  what  we  set  out  to  do.  Also,  if  we  are  not 
making  progress  toward  our  goal  we  want  to  know  sooner  rather  than  later  so 
we  can  change  our  strategy  and  begin  making  progress. 

To  gain  knowledge,  we  ask  questions.  The  best  evaluations  answer  simple 
questions  that  are  important  to  project  staff,  funders,  and  community  members. 
Chapter  1  outlined  how  community  coalitions  are  designed  to  reduce  risks  for 
CVD.  These  partnerships  mobilize  community  members  and  resources  to  make 
changes  in  multiple  sectors  of  the  community.  Community  changes  reduce  risks 
for  CVD  by  helping  community  members  adopt  healthy  behaviors.  How  do  you 
know  when  an  initiative  is  "working"?  Consider  posing  the  following  core 
evaluation  questions. 
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Core  evaluation  questions 

•  Was  the  community  mobilized  to  reduce  risks  for  CVD?  (Chapter  4) 

•  What  changes  in  the  community  resulted  from  the  initiative? 
(Chapter  4) 

•  Is  there  a  change  in  behavior  related  to  risks  for  CVD?  (Chapter  7) 

•  Does  the  initiative  have  a  community-level  outcome  related  to  risks  for 
CVD?  (Chapter  8) 

•  Is  community-level  outcome  related  to  changes  facilitated  by  the  initiative? 
(Chapters  4  and  8) 


Funders,  community  members,  and  staff  of  community  partnerships  may  have 
other  questions  about  the  initiative. 


Other  key  evaluation  questions 

•  Were  members  satisfied  with  the  partnership?  (Chapter  5) 

•  Were  the  community  changes  important  to  reducing  risks  for  CVD? 
(Chapter  6) 

•  What  critical  events  seemed  to  spur  rates  of  community  change? 
(Chapters  4  and  9) 


Evaluation  questions  are  designed  to  provide  information  on  whether 
community-based  programs  are  making  progress  toward  reducing  risks  for 
CVD.  Chapters  4-9  in  Part  II  describe  how  data  can  be  collected  to  answer 
evaluation  questions. 


When  deciding  what  questions  to  examine,  consider: 

•  What  members  and  leaders  of  the  community  partnership  want  to  know. 

•  Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

•  Resources  available  to  address  these  questions. 


14  CVD  Evaluation  Handbook 


As  recommended  in  the  Preface,  for  community  partnerships  with  HIGH 
resources  and  requirements  for  evaluation,  one  might  consider  addressing  all  of 
these  evaluation  questions  and  others  of  interest.  By  contrast,  for  community 
partnerships  with  LOW  resources  and  requirements,  one  might  address  a  subset 
of  questions  of  particular  interest. 
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A  Framework  for  Monitoring  and 
Improving  Cardiovascular  Disease 
Prevention  Initiatives 


This  chapter  presents  the  evaluation  framework  and  describes  information  that 
might  be  collected  to  answer  core  evaluation  questions  about  community 
partnerships  to  prevent  and  control  CVD.  Of  course,  the  information  actually 
obtained  should  depend  on  the  interests,  resources,  and  requirements  for  the 
evaluation. 

Before  the  evaluation  framework  is  presented,  a  note  on  terminology  may  be 
helpful.  The  language  and  vocabulary  of  evaluation  is  often  complex  and 
occasionally  inconsistent.  For  instance,  many  evaluators  trained  in  health 
education  use  the  term  process  to  refer  to  those  evaluation  activities  that 
examine  the  intervention  as  it  is  occurring,  impact  to  refer  to  the  direct  or 
immediate  effects  of  an  intervention,  and  outcome  to  refer  to  the  less  direct  or 
longer  term  effects,  such  as  changes  in  CVD  mortality.  On  the  other  hand,  for 
many  other  program  evaluators,  the  terms  impact  and  outcome  are  reversed, 
used  interchangeably,  or  modified  in  other  ways.  For  the  evaluation  system 
presented  in  this  manual,  we  refer  to  immediate  or  short-term  effects  as 
intermediate  outcomes  and  longer-term  effects  as  distal  outcomes.  This 
terminology  conveys  the  concept  that  changes  occur  over  time,  with 
intermediate  changes  generally  leading  to  changes  that  are  farther  along  or 
more  distal  to  the  initiation  of  programs.  Thus,  we  evaluate  the  sequence  of 
activities  that  may  be  of  interest  to  community-based  CVD  prevention  initiatives 
in  terms  of  process,  intermediate  outcomes,  and  distal  outcomes. 

This  system  for  evaluating  CVD  prevention  initiatives  is  linked  to  a  conceptual 
framework1  for  promoting  health  through  community  development.  We  have 
found  it  useful  to  describe  initiative  development  in  four  phases:  (a)  planning,  in 
which  a  vision,  mission  statement,  objectives,  strategies,  and  action  plans  are 
developed;  (b)  intervention,  in  which  staff  and  membership  take  action  in 
relevant  sectors  of  the  community  to  influence  the  behaviors  of  targets  and 
agents  of  change;  (c)  monitoring  of  intermediate  outcomes,  such  as  changes 
in  the  community  to  reduce  risks  and  enhance  protective  factors  for  CVD;  and 
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(d)  monitoring  of  more  distal  outcomes,  such  as  reported  physical  activity  and 
archival  records  of  tobacco  sales. 

The  top  part  of  Figure  3.1  shows  the  relationship  among  the  phases  and 
components  for  developing  community  partnerships.  Recommended  evaluation 
measures  for  each  phase  are  listed  below  the  diagram.2 


Figure  3.1 

Framework  for  evaluating  and 

improving  community  partnerships  to  prevent  CVD 


Planning 


o 


w 

Targets  of 

f 

w 

change 

y 

▼ 

▼ 

Intervention 

Risk/protective 
factors 

J 

< 

> 

W 

Agents  of 

cha 

ige 

^ 

Intermediate 
and  distal 
outcomes 


Intermediate 

Measurement 

Process 

Outcome 

Distal 

Instrument 

Measures 

Measures 

Outcome  Measures 

Monitoring  System 

1. 

Members  participating 

1.  Community 

(Chapter  4) 

2. 
3. 
4. 

5. 
6. 

7. 

Planning  products 
Media  coverage 
Financial  resources 
generated 
Dollars  obtained 
Services  provided 
Community  actions 

change 

Surveys  of 

8. 

Ratings  of  satisfaction 

2.  Ratings  of 

Satisfaction  and 

with  the  process  by 

significance  of 

Outcomes 

members 

outcomes  (i.e., 

(Chapters  5  and  6) 

community 
change) 

Behavioral  Surveys 

1 

Reported  behavior 

(Chapter  7) 

(e.g.,  tobacco  use, 
dietary  fat  intake, 
physical  activity) 

Community-level 

2. 

Archival  and  other 

Indicators 

records  (miles  of 

(Chapter  8) 

walking  trails,  no. 
of  businesses  with 
clean  air  policies, 
tobacco  sales) 

Interviews  with  Key 

3. 

Analysis  of  critical 

Participants 

events 

(Chapter  9) 

18 


CVD  Evaluation  Handbook 


The  evaluation  measures  are  described  in  terms  of  process,  intermediate,  and 
distal  outcomes.  Process  measures  correspond  to  the  planning  and 
intervention  phases  of  the  initiative;  these  measures  collect  data  on  whether  the 
initiative  completed  planning  and  is  operating  according  to  plans.  Intermediate 
outcomes  assess  changes  in  the  environment  corresponding  to  risk  reduction; 
these  measures  are  used  to  collect  data  on  whether  the  initiative  has  facilitated 
new  programs,  polices,  and  practices  (community  change)  consistent  with  the 
mission  of  preventing  CVD.  Distal  outcome  measures  correspond  to  whether 
the  initiative  has  changed  behavior  associated  with  risks  for  CVD  and  other 
community-level  indicators  associated  with  reduced  incidence  and  prevalence  of 
CVD. 

Table  3.1  summarizes  the  measurement  instruments  used  in  tracking  process, 
intermediate,  and  distal  outcomes.  The  following  sections  briefly  describe  each 
measurement  instrument,  the  types  of  information  collected  using  each 
instrument,  and  the  evaluation  question(s)  the  instrument  is  designed  to  address. 


Measurement  Instruments 

Monitoring  System 

The  monitoring  system  is  designed  to  provide  information  on  the  activities  of 
initiative  staff  and  members.  These  activities  are  analyzed  to  determine  which 
have  resulted  in  new  programs,  policies,  or  practices  (community  changes). 
Community  changes  and  other  project  activities  are  graphed  and  periodically 
provided  to  project  staff.  Community  leadership  collect  data  for  the  monitoring 
system,  and  they  also  communicate  the  data  to  their  membership  and  funding 
partners.  Measures  from  the  monitoring  system  can  help  answer  the  evaluation 
questions:  "Was  the  community  mobilized  to  address  the  mission?"  and 
"What  changes  in  the  community  resulted  from  the  initiative?" 
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Table  3.1 
System  for  evaluating  community  partnerships  to  prevent  CVD 


Measurement 
Instrument 


Measures  and 
Brief  Definitions 


Process  Evaluation 

Monitoring  System 


Member  Satisfaction 
Survey 


1.  Members  participating:  New  members,  affiliates,  or  partners  of 
the  initiative. 

2.  Planning  products:  New  objectives,  by-laws,  committees,  and 
other  internal  outcomes  resulting  from  planning  activities. 

3.  Media  coverage:  Instances  of  coverage  by  radio,  television,  and 
print  media  (e.g.,  newspapers  and  billboards). 

4.  Financial  resources  generated:  Instances  of  grants  received, 
donations,  in-kind  professional  services,  and  other  resources 
received  by  the  initiative. 

5.  Dollars  obtained:  Dollar  amount  of  grants  and  other  monies 
received  by  the  initiative. 

6.  Services  provided:   Classes,  workshops,  newsletters,  support 
groups,  screenings,  or  other  informational  or  service  programs 
provided  by  the  initiative  for  members  of  the  community. 

7.  Community  actions:  Actions  (e.g.,  phone  calls  and  personal 
contacts)  taken  outside  the  group  to  bring  about  changes  in  the 
community  related  to  the  initiative's  goals  and  objectives. 

8.  Satisfaction  with  aspects  of  the  initiative  (e.g.,  leadership) 
according  to  members. 


Intermediate  Outcome  Evaluation 


Monitoring  System 


Constituent  Survey  of 
Outcomes 


1.  Community  changes:  Changes  in  programs  (e.g.,  new  services 
established),  policies  (e.g.,  modified  city  ordinances),  and 
practices  (e.g.,  enhanced  enforcement)  of  agencies,  businesses, 
and  governmental  bodies  related  to  the  initiative's  goals  and 
objectives. 

2.  Significance  to  the  mission  of  community  changes  according  to 
members  of  the  initiative  and/or  outside  experts  in  prevention 
of  cardiovascular  diseases. 


Distal  Outcome  Evaluation 

Behavioral  Surveys  1.    Behavioral  measures  (e.g.,  diet,  physical  activity,  and  smoking). 
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Community-level  indicators  (e.g.,  archival  records  and  other 
indirect  or  global  measures  such  as  miles  of  walking  trails, 
percentage  of  stores  with  low-fat  milk,  and  tax  revenue  from 
tobacco  sales). 

Qualitative  information  about  critical  events  in  the  initiative's 
history  based  on  semistructured  interviews  with  key 
participants. 
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Monitoring  and  feedback  system 

Process  and  outcome  measures. 

Observational  system. 

Regular  feedback  on  performance. 


Process  and  Outcome  Measures.  Process  measures  collect  information  on 
whether  the  community  partnership  completes  planning  and  is  operating 
according  to  plans.  Outcome  measures  collect  information  on  whether  the 
initiative  is  making  progress  toward  reducing  risks  for  CVD.  Table  3.1  provides 
brief  definitions  of  eight  kinds  of  information  collected  by  the  monitoring 
system.  Process  measures  include  the  number  of  (a)  members  participating,  (b) 
planning  products,  (c)  instances  of  media  coverage,  (d)  financial  resources 
generated,  (e)  dollars  obtained,  (f)  services  provided,  and  (g)  community  actions. 
The  measure  of  intermediate  outcome  collected  using  the  monitoring  system  is 
community  change.  Please  review  carefully  the  measures  and  brief  definitions. 

Two  measures  in  the  monitoring  system — community  actions  and  community 
changes — may  be  especially  valuable  for  community  initiatives.  If  members  of 
the  partnership  meet  with  influential  community  members  or  agencies  (such  as 
the  food  service  staff  of  a  school  cafeteria),  they  can  produce  changes  in 
programs,  policies,  and  practices  (for  example,  menu  changes  to  reduce  fat  in 
school  lunches).  These  kinds  of  actions  and  outcomes  may  reduce  risks  and 
enhance  protective  factors  for  CVD  (for  example,  intake  of  dietary  fat  among 
school-aged  children).  High  rates  of  community  action  suggest  that  the  project 
is  attempting  to  create  a  large  number  of  changes  in  agencies  and  organi- 
zations. Rates  of  community  change  facilitated  by  the  partnership  provide  the 
earliest  evidence  that  the  group  is  on  the  road  to  reducing  risks  for  CVD. 

Collecting  Data.  Members  of  the  community  partnership,  usually  staff  or 
committee  chairpersons,  use  Event  Logs  to  record  monitoring  data.  Completed 
Event  Logs  provide  information  about  (a)  the  program  or  objective  for  which 
actions  were  taken,  (b)  actions  (what  was  done),  (c)  date  of  action  or  outcome 
(when),  (d)  target  of  action  (to  or  with  whom),  (e)  actors'  names  (by  whom),  (f)  the 
location  of  the  action  (where),  and  (g)  the  outcome  achieved  (change  in 
program,  policy,  or  practice).  The  logs  are  scored  by  those  responsible  for  the 
evaluation.  Evaluators  may  need  to  clarify  the  information  and  check  for 
completeness. 
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Those  responsible  for  the  evaluation  use  written  definitions  of  the  monitoring 
measures  to  categorize  activities  reported  in  the  logs.  (If  resources  permit, 
scoring  by  a  second,  independent  observer  permits  an  assessment  of  reliability 
or  interobserver  agreement.)  Once  the  activities  have  been  categorized, 
evaluators  count  up  the  number  of  activities  adhering  to  each  definition.  For 
example,  all  log  entries  that  report  a  change  in  a  program,  policy,  or  practice 
are  community  changes  and  are  tallied  to  provide  a  measure  of  how  much 
community  change  occurred  as  a  result  of  project  activities  during  the  reporting 
period. 

Feeding  Data  Back  to  Relevant  Audiences.  Those  with  evaluation 
responsibilities  graph  monitoring  data  and  use  the  graphs  to  provide  updates  on 
progress.  Data  are  graphed  and  provided  to  initiatives  on  a  regular  basis. 
Monitoring  data  enable  collaborators  to  detect  and  celebrate  early  successes, 
such  as  a  newly  established  program,  that  might  have  required  a  large  number 
of  actions.  Because  the  monitoring  system  measures  many  different  kinds  of 
activities,  the  partnership  will  be  able  to  see  how  its  members  and  staff  are 
expending  effort.  For  example,  a  community  coalition  may  be  providing  large 
amounts  of  information  and  referral  but  have  low  rates  of  community  action 
(meeting  with  community  leaders  or  agencies  to  create  community  change).  On 
the  basis  of  this  feedback,  the  coalition  may  decide  to  redirect  efforts  to  increase 
rates  of  community  action  and  change. 

Partnership  staff  can  use  these  data  in  newsletters  and  reports  and  in  meetings 
with  members  and  potential  funders  to  promote  the  initiative  or  solidify  support. 
Chapter  4  describes  the  monitoring  system  in  more  detail. 

Surveys  of  Satisfaction  and  Outcome 

Surveys  help  assess  satisfaction  with  the  coalition's  operations  (process)  and 
achievements  (intermediate  outcome)  from  the  perspective  of  members  and 
other  important  audiences.  We  recommend  that  evaluators  provide  reports  to 
leadership  and  membership  of  the  initiative  about  members'  satisfaction  with  the 
initiative-building  process  and  whether  achievements  of  the  partnership  are 
likely  to  reduce  risks  for  CVD. 


Constituent  surveys 

•  Satisfaction  with  process. 

•  Significance  of  outcomes. 
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Survey  of  Member  Satisfaction.  At  the  end  of  each  operating  year,  we 
recommend  conducting  a  survey  to  assess  members'  satisfaction  with  the 
community  initiative.  Chapter  5  describes  this  Member  Satisfaction  Survey  in 
more  detail.  The  satisfaction  survey  can  help  answer  the  evaluation  question: 
"Were  members  satisfied  with  the  initiative?" 

Survey  of  Outcomes.  A  survey  can  be  used  to  assess  the  significance  of 
community  changes  resulting  from  the  initiative's  efforts.   Chapter  6  provides 
more  information  on  the  survey  of  outcomes.  The  Outcome  Survey  can  help 
answer  the  evaluation  question:   "Were  the  community  changes  important  to 
reducing  risks  for  CVD?" 


Behavioral  Surveys 

To  help  guide  efforts  and  to  assess  distal  outcomes  of  the  community 
partnership,  it  may  be  appropriate  to  obtain  and  analyze  existing  data  on  CVD 
risk  behaviors.  For  example,  relevant  modules  of  the  Behavioral  Risk  Factor 
Surveillance  System  (BRFSS)  questionnaire  can  provide  important  data  on 
whether  citizens  are  changing  diet,  exercise,  or  tobacco-related  behaviors  as  a 
result  of  initiative  efforts.  Chapter  7  provides  more  information  about  using 
behavioral  surveys.  With  adequate  resources,  behavioral  surveys  can  help 
answer  the  evaluation  question:  "Is  there  a  change  in  behavior  related  to  risks 
for  CVD?" 


Community-level  Indicators 

We  also  recommend  using  archival  records  and  other  indirect  or  global 
measures  to  select  and  obtain  community-level  indicators  of  whether  risks  of 
CVD  have  been  reduced.  For  example,  for  tobacco  control  these  might  include 
tax  data  on  per  capita  consumption  of  cigarettes  and  smokeless  tobacco.  If  tax 
revenues  decline,  it  may  suggest  that  the  partnership  is  having  an  impact  on  a 
very  important  risk  factor  for  CVD.  Chapter  8  discusses  how  to  use  community- 
level  indicators  in  detail.  Data  from  community-level  indicators  can  help  answer 
the  evaluation  question:    "Does  the  initiative  have  a  community-level 
outcome  related  to  risks  for  CVD?" 


Interviews  with  Key  Participants 

Finally,  we  recommend  using  interviews  with  active  leaders  of  the  community 
partnership  to  provide  information  about  the  initiative's  efforts,  achievements, 
and  barriers  to  success.  The  interviews  are  usually  conducted  several  years  into 
the  implementation  of  the  initiative  or  during  transitions  between  leaders.  The 
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focus  of  the  interviews  is  to  identify  and  analyze  critical  events  in  the  history  of 
the  partnership,  such  as  securing  the  initial  grant  or  action  planning.  Chapter  9 
offers  items  about  how  to  identify  and  explore  critical  events  in  interviews  with 
key  participants.  Interviews  with  key  participants  can  help  answer  the  evaluation 
question:   "What  critical  events  seemed  to  spur  rates  of  community  change?" 


Conclusion 

The  monitoring  system  tracks  many  process  and  outcome  measures.  It  may  be 
the  most  important  part  of  the  evaluation,  because  it  collects  data  on  the  short- 
term  effects  of  initiative  efforts  (community  actions  and  changes).  Through  the 
monitoring  system,  the  initiative  gets  quick  feedback  on  how  well  its  efforts  are 
paying  off  in  terms  of  community  change.  Other  aspects  of  the  evaluation  use 
surveys  and  a  variety  of  other  reporting  instruments  and  are  designed  to  assess 
member  satisfaction  with  the  process  and  outcomes  of  the  initiative,  identify 
behavior  change  as  a  result  of  coalition  efforts,  determine  whether  the  initiative 
is  having  an  impact  on  community-level  indicators,  and  identify  critical  events  in 
the  history  of  the  partnership.  The  following  six  chapters  present  detailed 
descriptions  of  each  of  the  six  core  measurement  instruments. 


When  deciding  what  measurement  instruments  to  use,  consider: 

•  What  members  and  leaders  of  the  community  partnership  want  to  know. 

•  Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

•  Resources  available  to  address  these  questions. 


For  community  partnerships  with  HIGH  resources  and  requirements  for 
evaluation,  consider  using  all  of  these  tools  for  evaluating  community 
partnerships  and  others  of  value.  This  might  presume  the  availability  of  coalition 
staff  to  provide  information,  a  person  with  basic  skills  in  evaluation,  and  a 
modest  budget  specifically  for  expenses  related  to  the  evaluation. 

For  community  partnerships  with  LOW  resources  and  requirements,  consider 
using  only  a  few  of  these  measurement  instruments.  This  might  presume  little  or 
no  evaluation  budget  and  virtually  no  reporting  requirements.  Staff  might  be 
available  for  evaluation  activities  for  several  hours  a  week  or  less,  with  minimal 
training  in  evaluation.  Recommended  instruments  to  be  used  by  community 
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partnerships  with  minimal  resources  and  requirements  follow.  This  list  may  be 
modified  based  on  the  community  partnership's  interests,  requirements,  and 
resources  for  evaluation. 


Recommended  Measurement  Instruments  and  Time  Estimates 

•  Monitoring  system  (Chapter  4):    Approximately  15  hours  per  month  if 
only  community  change  is  measured  on  an  ongoing  basis. 

•  Survey  of  satisfaction  (Chapter  5):  Approximately  10  hours  annually. 

•  Survey  of  outcomes  (Chapter  6):  Approximately  15  hours  annually. 

•  Behavioral  survey  (Chapter  7):  Approximately  20  hours  if  data  are 
available. 

•  Community-level  indicators  (Chapter  8):  Approximately  20  hours,  only  if 
data  are  available  from  other  sources,  annually. 

•  Interviews  with  key  participants  (Chapter  9):  Approximately  30  hours 
every  2  years. 


*Based  on  experience  with  a  variety  of  partnerships  for  community  health  and  development. 
Time  estimates  may  vary  depending  on  the  need  for  negotiation  of  instruments,  number  of 
individuals  you  wish  to  collect  data  from,  technology  available  to  analyze  data,  reporting 
requirements,  and  experience  implementing  data  collection  systems. 
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PART  II 


The  Evaluation  System 


What's  Ahead . . . 

♦  Monitoring  System 

♦  Assessing  Member  Satisfaction 

♦  Assessing  the  Importance  of  Outcomes 

♦  Using  Behavioral  Data  and  Surveys 

♦  Community-level  Indicators  of  Cardiovascular 
Health 


*•    Conducting  Interviews  with  Key  Participants 


♦> 


Process 
Measure 

Intermediate 
Outcome 
Measure 

Distal 
Outcome 
Measure 

4 


Monitoring  System 


Monitoring  data  contribute  information  about  the  process  and  intermediate 
outcomes  of  community  partnerships.  Logs  and  interviews  with  those  most 
involved  in  the  partnership  are  used  to  gather  data.  Several  core  evaluation 
questions  may  be  addressed  using  data  from  the  monitoring:  (a)  Was  the 
community  mobilized  to  reduce  the  risks  for  CVD?  (b)  What  changes  in  the 
community  resulted  from  the  initiative?  (c)  What  critical  events  seemed  to  spur 
rates  of  community  change?  and  (d)  Are  community-level  outcomes  related  to 
changes  facilitated  by  the  initiative? 


Monitoring  system:  What  and  why? 

What  is  it? 

•  A  way  of  tracking  major  events  and  accomplishments. 

Why  use  it? 

•  To  understand  the  initiative. 

•  To  decide  where  to  focus  efforts. 

•  To  promote  awareness  of  accomplishments. 

•  To  recruit  support. 

•  To  secure  grants. 


Monitoring  data  help  detect  how  initiatives  use  resources.  For  instance,  if  the 
purpose  of  the  coalition  is  to  serve  as  a  catalyst,  high  rates  of  community 
change,  rather  than  provision  of  direct  services,  may  be  particularly  valued. 
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Monitoring  data  also  point  out  the  community  sectors  that  are  more  engaged. 
For  example,  are  most  changes  occurring  in  worksites  with  fewer  changes  in 
schools?  These  data  enable  partnerships  to  determine  which  strategies,  risks, 
and  protective  factors  are  being  addressed.  For  example,  for  a  tobacco  control 
effort,  is  lots  of  information  being  provided  with  few  changes  that  modify  access 
to  products  such  as  tobacco?  Staff  and  leadership  can  use  monitoring  data  to 
promote  community  awareness  of  the  initiative's  early  successes,  recruit 
community  support,  and  secure  financial  resources. 

To  monitor  the  major  events  and  accomplishments  of  the  collaborative 
partnership,  use  recording  sheets,  called  log  forms,  and  interviews  with  people 
completing  the  monitoring  log  forms.  Although  these  will  vary  with  each 
initiative,  important  events  may  include  services  provided;  instances  of  media 
coverage;  resources  generated;  community  actions;  and  changes  in  programs, 
policies,  and  practices  (community  changes)  related  to  the  mission  of  reducing 
the  risks  for  CVD. 


Monitoring  System  —  The  Five  Steps 

This  chapter  describes  how  to  use  the  monitoring  system  to  collect  important 
events  and  other  data  relevant  to  community  initiatives  for  reducing  the  risks  for 
CVD. 


Monitoring  system:   How  to's 

Complete  log  forms. 

Clarify  log  entries. 

Categorize  log  entries. 

Assess  reliability  of  scores  (optional). 

Graph  data  and  provide  feedback. 


There  are  five  major  steps  in  the  monitoring  process.   First,  key  participants, 
such  as  active  members  of  the  coalition  and  project  staff,  describe  the  project's 
activities  on  record  sheets  (log  forms).  Second,  the  entries  are  clarified,  if 
necessary.  Third,  someone  scores  the  reported  activities  into  different 
categories  using  a  set  of  definitions.  Fourth,  someone  else  also  categorizes  the 
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reported  activities  to  obtain  an  estimate  of  the  accuracy  or  reliability  of  the  first 
categorization  (optional).  Finally,  the  cumulative  number  of  events  from  each 
different  category  is  graphed.  The  graphs  are  then  shared  with  key  participants 
and  the  community  at  large.  The  next  sections  outline  these  five  steps. 

Keep  in  mind  that  monitoring  is  one  of  the  most  complicated  processes 
described  in  this  handbook.  The  last  three  steps  (categorizing  log  entries, 
assessing  reliability,  and  graphing  and  providing  feedback)  may  require 
additional  training  to  complete.  Those  responsible  for  monitoring  coalition 
activities  may  consult  the  background  information  in  Appendix  A  for  additional 
instructions,  examples,  and  practice  exercises.  The  background  information  and 
materials  provide  more  detailed  instruction  for  using  the  monitoring  system 
outlined  in  the  following  five  steps. 

Step  1 :  Complete  Log  Forms 

Materials  needed:  Blank  log  forms  (in  Appendix  A) 

Time  required:  Up  to  2  hours  per  week 

Who  does  it?  Active  members  of  the  partnership  and  staff 

Information  to  complete  log  forms  can  be  gathered  in  a  number  of  ways.  Many 
community  partnerships  have  staff  and  active  members  complete  log  forms  on  a 
monthly  basis.  Some  groups  complete  log  forms  during  committee  or  staff 
meetings — log  forms  then  serve  as  meeting  notes  as  well  as  provide  information 
for  the  evaluation.  Some  staff  members  complete  log  forms  by  reviewing  their 
calendars,  meeting  minutes,  and  newspaper  articles  that  report  on  the  group's 
activities. 

A  number  of  different  log  forms  have  been  developed  to  gather  different  types 
of  information.  Event  Logs  are  used  to  record  activities  that  have  the  goal  of 
making  systems  or  community  changes.  Community  changes  are  changes  in 
programs,  policies,  or  practices  designed  to  reduce  the  risks  for  CVD. 
Individuals  completing  the  Event  Logs  are  asked  to  describe  the  event  in  detail, 
including  providing  information  on  (a)  why  the  event  is  important,  (b)  what 
happened  as  a  result  of  the  event,  (c)  who  was  involved,  (d)  what  organizations 
were  collaborators,  (e)  what  objective  or  sector  of  the  community  the  event  is 
related  to,  and  (f)  whether  this  was  the  first  time  the  event  happened.  Answers 
to  these  questions  help  determine  what  category  the  event  fits  into  (see  Step  3: 
Categorize  Log  Entries). 

Many  initiatives  want  information  about  other  important  activities  of  the  group. 
The  Ongoing  Services  Provided  Log  gathers  information  about  educational  or 
service  programs  provided  by  the  group.  Smoking  cessation  classes, 
presentations  describing  the  project,  and  mall  walking  clubs  established  for 
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seniors  in  the  community  are  all  examples  of  services  that  could  be  provided  by 
an  initiative  to  reduce  risks  for  CVD.  Many  groups  are  interested  in  tracking 
how  much  media  attention  they  receive,  which  is  the  purpose  of  the  Media 
Coverage  Log.  The  Resources  Generated  Log  tracks  both  in-kind  and 
monetary  resources  secured  by  the  initiative. 

Four  log  forms  have  been  described:  Event  Log,  Ongoing  Services  Provided 
Log,  Media  Coverage  Log,  and  Resources  Generated  Log.  Examples  of  these 
forms  may  be  found  in  Figures  4.1-4.4.  These  sample  logs  have  been  filled  out 
with  examples  of  events,  ongoing  services,  media,  and  resources  obtained  to 
show  how  each  could  be  used  by  a  CVD  prevention  effort. 

Why  are  there  so  many  log  forms?  Each  of  the  forms  is  designed  to  collect 
information  that  is  needed  to  categorize  the  partnership's  activity  according  to 
a  set  of  definitions.  The  categorized  information,  rather  than  an  unorganized 
list  of  the  group's  activities,  is  more  manageable.  It  helps  answer  core 
evaluation  questions  more  efficiently. 

Step  2:  Clarify  Log  Entries 

Materials  needed:  Completed  log  forms 
Time  required:  Up  to  1  hour  per  month 
Who  does  it?  Project  staff,  internal  or  external  evaluators 

Log  entries  are  sometimes  incomplete  or  require  additional  explanation. 
Reported  events  should  be  complete  enough  so  that  someone  who  is  unfamiliar 
with  the  organization  could  understand  them.  Clarifying  logs  often  requires 
conversation  (in  person  or  through  phone  calls)  between  the  person  who 
completed  the  log  form  and  the  person  who  is  going  to  categorize  the  activities 
reported  in  the  log.  This  step  is  especially  important  if  someone  external  to  the 
project,  such  as  outside  evaluators,  categorizes  the  log  entries. 
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Step  3:  Categorize  (Score)  Log  Entries 

Materials  needed:  Completed  and  clarified  log  forms 

Time  required:    1-2  hours  per  month 

Who  does  it?  Project  staff,  internal  or  external  evaluators 

Additional  instructions,  examples,  and  practice  exercises  are  provided 
in  the  background  information  in  Appendix  A 

This  step  is  the  most  important,  and  most  difficult,  part  of  the  monitoring 
system.   In  this  step,  definitions  are  used  to  categorize  log  entries.   Entries  on 
the  log  forms  are  compared  to  definitions  and  categorized  according  to  which 
definition  best  describes  the  event.  Scoring  can  be  done  by  a  project  staff 
member,  internal  evaluator,  or  external  evaluator. 

The  categories  for  scoring  are  as  follows:   Community  actions,  community 
changes,  planning  products,  services  provided,  media  coverage,  and 
resources  generated.    Brief  definitions  of  each  type  of  event  are  provided  in 
Table  4.1.  Complete  instructions  for  coding  and  expanded  definitions  are  given 
in  the  background  information  in  Appendix  A. 

Scoring  log  entries  is  sometimes  difficult  and  requires  practice.  Using  log  forms, 
such  as  the  Event  Log  provided  at  the  end  of  this  chapter,  is  helpful  because  the 
form  requests  information  needed  to  assign  a  category  to  each  event.  Entries  in 
the  Event  Log  typically  will  be  assigned  the  scoring  category  of  community 
action,  community  change,  or  planning  products.  Scores  for  entries  on  the 
Ongoing  Services  Provided  Log,  Media  Log,  and  Resources  Obtained  Log  will 
often  be  more  straightforward.  Entries  should  be  reviewed  carefully,  however, 
to  make  sure  they  match  the  definitions  of  services  provided,  media  coverage, 
resources  generated,  and  other. 
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Table  4.1 

Types  of  community  partnership  activities, 

their  definitions,  and  examples  of  log  entries 


Type  of 

partnership 

activity 


Definition 


Example 


Community  Actions  taken  in  the  corn- 

Actions  munity  to  bring  about  a  new 

or  modified  program,  policy, 
or  practice  to  reduce  risks 
for  CVD 


Memorandum  of  agreement  between 
the  Church  League  and  City  Parks  and 
Recreation  Department  to  sponsor 
summer  fitness  clubs  for  adolescents 


Community 
Changes 


Planning 
Products 


Services 
Provided 


New  or  modified  programs, 
policies,  or  practices  in  the 
community  facilitated  by  the 
initiative  that  reduce  risks 
for  CVD 


The  results  or  products  of 
planning  activities  with- 
in the  group 


Events  that  are  designed  to 
provide  information  or 
instruction  or  to  develop  skills 
of  people  in  the  community 


Programs:  Established  new  physical 
fitness  programs  for  older  adults 
(physical  activity);  all  county  physicians 
and  other  health  care  providers  began 
assessing  and  counseling  all  patients  on 
smoking,  diet,  and  physical  activity 
(tobacco,  diet,  physical  activity) 

Policies:  Adopted  ordinance  that  would 
rescind  license  of  merchants  that  sell 
tobacco  to  minors  (tobacco);  adopted 
policy  requiring  lower  fat  school  lunches 
(diet);  established  a  policy  allowing 
flexible  work  hours  to  accommodate 
exercise  (physical  activity) 

Practices:  Businesses  refused  to  provide 
free  samples  of  tobacco  (tobacco);  local 
media  provided  free  advertisement  for 
restaurants  that  offer  lower  fat  entrees 
(diet);  parks  and  recreation  department 
offered  activities  for  youth  and  families 
of  low  income  (physical  activity) 

Hiring  of  staff;  mission  statement 
developed;  strategic  plan  adopted; 
guidelines  developed  for  awarding 
minigrants;  committee  formed;  grant 
application  completed 

Class  held;  workshop  conducted; 
presentation  to  Rotary;  displays  of 
project  at  4-H  fair,  media  campaign 
using  physical  activity  guidelines 
recommended  by  CDC 
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Table  4.1 

Types  of  community  partnership  activities, 

their  definitions,  and  examples  of  log  entries  (Continued) 


Type  of 

partnership 

activity 


Definition 


Example 


Media 

Coverage  of  the  initiative  or 

Coverage 

its  projects  in  the  newspaper, 

radio  or  television,  or 

newsletter 

Resources 

Acquisition  of  funding  for  the 

Generated 

initiative  through  grants, 

donations,  or  gifts  in-kind 

KJHK  aired  30  public  service 
announcements  (PSAs)  (15  minutes 
total)  describing  risk  factors  for  CVD 


$2,500  of  donated  goods  secured  for 
Health  March  and  Rally;  $10,000  grant 


This  method  of  categorizing  log  entries  is  actually  a  measurement  system.   It 
allows  community  partnerships  to  sort  their  efforts  into  types  of  activities.  Once 
actions  are  categorized  (Step  3)  and  graphed  (Step  5),  the  initiative  can  get  a 
clearer  idea  of  how  much  effort  it  is  expending  in  different  types  of  activities. 


Step  4  (optional):  Assessing  Reliability  (Agreement  on  Scoring) 

Materials  needed:  Scored  log  forms,  formula  for  calculating  reliability 
(Appendix  A)  or  computer  program  for  calculating  reliability 

Time  required:    1-2  hours  per  month 

Who  does  it?  Project  staff,  internal  or  external  evaluators 

Additional  instructions,  examples,  and  practice  exercises  are  provided 
in  Appendix  A 

Some  people  will  be  skeptical  of  the  codes  assigned  to  the  log  entries.  How  do 
they  know  the  entries  were  put  in  the  correct  categories?  To  address  these 
concerns,  another  person  can  score  the  same  set  of  logs.  The  two  scores  are 
compared,  thus  providing  an  estimate  of  reliability.  Reliability  scores  can  be 
obtained  by  calculating  with  a  computational  formula  manually  or  by  using  a 
statistics  computer  program.  Complete  instructions  for  assessing  reliability  are 
provided  in  the  background  information  section  in  Appendix  A.   Partnerships 
with  minimal  resources  and  requirements  for  evaluation  might  not  be  expected 
to  assess  reliability. 
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Step  5:  Graphing  and  Using  the  Data 

Materials  needed:  Scored  log  forms,  data  from  previously  completed  logs, 
graph  paper  or  computer  graphing  program 

Time  required:    1-3  hours  per  month 

Who  does  it?  Project  staff,  internal  or  external  evaluators 

Additional  instructions,  examples,  and  practice  exercises  are  provided 
in  Appendix  A 

Filling  out  and  scoring  logs  takes  time.  What  are  the  results?  Once  log  items  are 
categorized  or  scored,  the  number  of  items  in  each  category,  such  as 
community  change,  community  action,  or  planning  products,  can  be  graphed. 
Data  presented  in  this  way  will  help  projects  compare  how  much  effort  they  are 
spending  in  different  areas.  The  graph  can  be  shared  monthly  or  quarterly  with 
project  staff,  leadership,  membership,  funders,  and  partners  of  the  initiative.  A 
sample  graph  is  shown  in  Figure  4.5.  Instructions  for  constructing  graphs  and 
providing  feedback  on  the  data  are  included  in  Appendix  A. 
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Figure  4.5 
Sample  feedback  graph  of  community  changes 
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Community  changes  are  new  programs,  policies,  or  practices  in  the 
community  that  are  produced  by  project  members  and  are  relevant  to 
the  initiative's  goals. 

Examples  of  community  changes  produced  by  the 
county  include: 

•  Comprehensive  tobacco  control  curriculum  implemented  in  YWCA 
after-school  programs. 

•  Cooks  at  Happy  Corner  day  care  were  trained  in  preparing  menus 
and  foods  to  meet  nutritional  guidelines. 

•  The  company,  PackSac,  added  a  gym  to  its  new  plant  and  began 
offering  incentives  for  employees  to  stay  fit. 

Patterns  to  notice: 

There  has  been  a  steady  rate  of  community  changes  produced  since 
January  1994  when  the  director  was  hired.  Four  new  programs  were 
established  in  February  1995. 
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Adjusting  the  System  to  Meet  Your  Needs 


When  deciding  what  data  to  collect  with 
the  monitoring  system,  consider: 

•  What  members  and  leaders  of  the  community  partnership  want  to  know. 

•  Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

•  Resources  available  to  address  these  questions. 


The  amount  of  time  and  resources  needed  to  monitor  a  CVD  prevention  effort 
will  vary  depending  upon  a  variety  of  factors.   Of  course,  monitoring  activities 
can  be  prioritized  and  tailored  to  match  the  requirements  of  the  evaluation  and 
resources  available  it  carry  it  out. 


Recommended  Monitoring  System  for  Those  Operating 
on  a  Shoestring  Budget 

Collect,  graph,  and  provide  quarterly  feedback  on  community  change  only 
(using  the  Event  Log  only). 
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Assessing  Member  Satisfaction 


Surveys  can  be  used  to  assess  satisfaction  of  members  of  the  coalition  with  the 
functioning  of  the  partnership.  These  surveys  help  examine  the  process  of  the 
initiative,  an  important  aspect  of  the  framework  for  evaluation  described  in 
Chapter  3.  Specific  survey  questions  may  address  a  variety  of  areas,  including 
planning,  leadership,  services,  community  involvement,  and  progress  toward 
accomplishing  goals.  Such  surveys  help  address  a  key  evaluation  question: 
"Were  constituents  satisfied  with  the  initiative?" 


Survey  of  satisfaction:    What  and  why? 


What  is  it? 


•  A  survey  to  assess  members'  satisfaction  with  the  functioning  of  the 
initiative. 

Why  do  it? 

•  To  inform  the  partnership  about  strengths  and  weaknesses  from  the 
perspective  of  coalition  members  or  constituents. 

•  To  identify  issues  that  could  become  potential  problems. 


The  satisfaction  survey  allows  members  of  the  partnership  to  give  feedback  to 
leadership  on  how  well  the  initiative  is  doing.   Data  from  the  survey  helps 
identify  strong  and  weak  features  of  the  initiative.   The  survey  also  permits 
issues  to  be  dealt  with  early  on  before  they  escalate  into  problems. 

This  section  describes  how  to  obtain  ratings  of  members'  satisfaction  with  the 
initiative  by  using  the  survey. 
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Satisfaction  survey:     How  to's 

Develop  a  survey  to  assess  member  satisfaction  with: 

•  Planning  and  implementation. 

•  Leadership. 

•  Services. 

•  Community  involvement  with  the  initiative. 

•  Progress  and  outcome. 

Obtain  ratings  from  members  of  the  coalition  or  project. 
Use  the  data  to  improve  the  functioning  of  the  initiative. 


Those  responsible  for  the  evaluation  help  design  the  survey  process  in 
collaboration  with  leadership  of  the  partnership.  Questionnaires  are  distributed 
(usually  by  mail)  to  members  of  the  initiative.  Completed  questionnaires  are 
returned  by  mail  (or  drop  box)  to  preserve  the  members'  anonymity.  Data  from 
questionnaires  are  entered  and  summarized.  Results  are  provided  to  the  staff 
and  membership  of  the  coalition  or  project. 

This  survey  should  be  done  annually.  The  entire  process  from  beginning  to  end 
usually  takes  approximately  three  months.   Leadership  of  the  project  spend 
approximately  five  hours  or  less  developing  questions  and  distributing  the 
questionnaires. 


Adjusting  the  System  to  Meet  Your  Needs 

A  complete  generic  satisfaction  survey  for  a  coalition  to  reduce  risks  for  CVD  is 
at  the  end  of  this  chapter.  A  sample  cover  letter  and  a  sample  memorandum  of 
results  are  also  included. 
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When  deciding  what  to  ask  members  about,  how  often  to  ask,  and 
by  what  means  to  gather  information,  consider: 

•  What  members  and  leaders  of  the  community  partnership  want  to  know. 

•  Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

•  Resources  available  to  address  these  questions. 


For  community  partnerships  with  HIGH  resources  and  requirements  for 
evaluation,  consider  doing  annual  (or  even  more  frequent)  assessments  of 
member  satisfaction.  This  might  presume  the  availability  of  coalition  staff  to 
print  and  distribute  questionnaires;  a  person  with  basic  skills  in  data  entry  and 
analysis;  and  a  modest  budget  for  printing,  distribution,  data  analysis,  and  other 
related  expenses. 

For  community  partnerships  with  LOW  resources  and  requirements,  consider 
assessing  member  satisfaction  informally  or  less  often.  This  might  presume  little 
or  no  evaluation  budget  and  virtually  no  reporting  requirements.  Staff  might  be 
available  for  one  day  or  less,  with  minimal  secretarial  support  for  data  entry. 

Recommended  Strategies  for  Assessing  Member  Satisfaction 

on  a  Shoestring  Budget 

•  Distribute  surveys  at  coalition  meetings.  Use  a  calculator  to  summarize  the 
data  on  member  satisfaction.  If  time  allows,  review  and  summarize  the  data 
at  the  end  of  the  meeting. 

•  Include  a  limited  number  of  questions  in  a  coalition  newsletter  or  other 
planned  mailing.  Ask  members  to  donate  the  returned  postage. 

•  Collaborate  with  a  local  community  college  or  university  that  conducts  the 
survey  as  part  of  a  class  project. 

•  Instead  of  using  a  survey,  collect  information  in  informal  listening  sessions. 
Flip  charts,  a  writing  board,  or  chalk  board,  or  plain  paper  can  be  used  to 
capture  member  input  on: 

Issues 
Barriers 
Resources 
Alternative  solutions. 

•  Local  organizations,  such  as  the  League  of  Women's  Voters  or  Junior 
League,  might  facilitate  these  kinds  of  listening  sessions  free  of  charge. 
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Materials  for  Assessing  Member  Satisfaction 

The  following  section  provides  a  sample  member  satisfaction  survey  and  other 
background  information.   Materials  that  can  be  tailored  to  suit  any  CVD 
prevention  effort  planning  to  do  a  Member  Satisfaction  Survey  are  included. 


The  following  are  included 

A  sample  cover  letter  for  a  satisfaction  survey  (Figure  5.1)  that  describes  the 
survey's  purpose,  how  it  should  be  filled  out,  and  what  will  be  done  with  the 
results. 

A  format  for  a  generic  survey  (Figure  5.2)  outlining  the  components  and 
suggested  format  that  can  be  used  as  a  guide  when  developing  a  member 
satisfaction  survey. 

An  example  memorandum  (Figure  5 . 3)  that  describes  how  the  results  of  the 
member  satisfaction  survey  can  be  summarized  and  presented.  This 
document  may  be  used  by  a  partnership  to  communicate  information 
about  the  results. 
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Figure  5.1 
Sample  cover  letter  for  the  Member  Survey  of  Satisfaction 


Date 


Dear  Member  of  (the  initiative): 

The  purpose  of  the  attached  member  satisfaction  survey  is  to  get  your  feedback 
on  how  well  (the  initiative)  is  doing  in  implementing  its  goals.  As  you  know,  (the 
initiative's)  mission  is  to in  (city /county /state) . 

Please  complete  each  question  by  circling  the  number  that  best  shows  your 
satisfaction  with  that  aspect  of  the  initiative.  Your  comments  and  suggestions 
on  how  to  improve  (the  initiative)  are  welcome. 

To  protect  anonymity,  use  the  enclosed  envelope  to  return  your  completed 
questionnaire  to  our  evaluators,  the  (name  them). 

Thanks  in  advance  for  your  valuable  time  and  feedback. 


Best  regards, 


Executive  Director 


Enclosure 
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Figure  5.2 
Generic  Member  Survey  of  Satisfaction 


PLEASE  RETURN  TO 


(return  address) 


PLEASE  RETURN  BY 


(due  date) 


(Date) 

We  appreciate  your  comments  on  how  well  (the  initiative)  is  doing.  Please 
provide  your  suggestions  for  improving  the  initiative  and  describe  the  things 
with  which  you  are  most  pleased.  For  each  item,  please  circle  the  number  that 
best  shows  your  satisfaction  with  that  aspect  of  the  initiative.      Provide 
additional  comments  if  you  wish. 


Planning  and  Implementation: 

1 .  Planning  process  used  to  prepare  the 
objectives  for  the  coalition. 

2.  The  objectives  selected. 

3.  Follow  through  on  activities. 

4.  Competence  of  staff. 
Comments: 


Very 
dissatisfied 

Very 
satisfied 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

ea 

5. 

dership: 

Clarity  of  the  vision  for  where  the 
initiative  should  be  going. 

1 

2 

3 

4 

5 

6. 

Strength  and  competence  of 
leadership. 

1 

2 

3 

4 

5 

7. 

Sensitivity  to  cultural  issues. 

1 

2 

3 

4 

5 

8. 

Use  of  the  media  to  promote 
awareness  of  the  project's  goals, 
actions,  and  accomplishments. 

1 

2 

3 

4 

5 

9. 

Opportunities  for  members  to  take 
leadership  roles. 

1 

2 

3 

4 

5 

Comments: 
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Very  Very 

Services:  dissatisfied  satisfied 


10.  Quality  of  the  services  provided,  12  3  4  5 
sponsored,  or  initiated  by  the 

initiative. 

11.  Quantity  of  the  services  provided,  12  3  4  5 
sponsored,  or  initiated  by  the 

initiative. 

12.  Training  and  technical  assistance.  12  3  4  5 

13.  Fund-raising  and  grantwriting.  12  3  4  5 

14.  Information  and  referral.  12  3  4  5 

15.  Advocacy.  12  3  4  5 
Comments: 


Community  Involvement 
in  the  Initiative: 

16.  Extent  to  which  members  of  the  12  3  4  5 
community  participated  in  the 

planning. 

17.  Extent  to  which  members  of  the  12  3  4  5 
community  participated  in  the 

implementation. 

18.  Extent  to  which  members  of  the  12  3  4  5 
community  participated  in  the 

programs. 

19.  Participation  of  influential  people  12  3  4  5 
from  key  sectors  of  the  community. 

20.  Participation  of  people  at  most  risk  12  3  4  5 
for  CVD. 

21.  Diversity  of  membership  of  the  12  3  4  5 
initiative. 

Comments: 
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Very  Very 

Progress  and  Outcome:  dissatisfied  satisfied 


22.  Progress  in  meeting  the  objectives.  12  3  4  5 

23.  Success  in  generating  resources.  12  3  4  5 

24.  Fairness  with  which  funds  and  12  3  4  5 
opportunities  are  distributed. 

25.  The  initiative's  contribution  to  the  goal  12  3  4  5 
of  reducing  intake  of  dietary  fat. 

26.  The  initiative's  contribution  to  the  goal  12  3  4  5 
of  increasing  physical  activity. 

27.  The  initiative's  contribution  to  the  goal  12  3  4  5 
of  preventing/reducing  tobacco  use. 


28.     The  initiative's  contribution  to  the  goal 
of  (list  other  major  goals  if 
appropriate). 

Comments: 


Overall  Approval  Rating: 

29.     Is  the  community  better  off  today  Yes  No 

because  of  (the  initiative)? 
(Circle  one) 

Overall  Comments: 
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Figure  5.3 

Sample  memorandum  of  results  for  the 

Member  Survey  of  Satisfaction 


Date: 


To:        (Leadership  and  Steering  Committee) 

From:  The  Evaluation  Team 

Re:       Results  of  the  Annual  Member  Satisfaction  Survey  for  (the  initiative). 

Enclosed  please  find  the  results  of  the  Annual  Member  Satisfaction  Survey.  We 

had  a  good  response — X  of  X  people  ( %)  completed  the  survey.  This  report 

includes  the  average  rating  and  the  range  of  responses  for  each  question,  and  a 
summary  of  the  comments  for  each  section. 

Generally,  those  members  who  responded  were  satisfied  with  many  aspects  of 
coalition  functioning  outlined  in  the  survey.  The  range  of  responses  was  from  3 
to  5  for  each  issue.  The  highest  ratings  were  noted  for  the  strength  of 
competence  of  staff,  the  strength  and  competence  of  leadership,  and  the  clarity 
of  the  vision  for  the  initiative.  The  lowest  rating  was  noted  for  item  25,  the 
group's  contribution  to  the  goal  of  reducing  intake  of  dietary  fat.  Ninety-three 
percent  of  the  respondents  felt  that  the  community  was  better  off  today  because 
of  (the  initiative). 

Many  of  the  respondents  also  provided  comments  at  the  end  of  each  section. 
The  comments,  in  addition  to  the  ratings  of  each  question,  may  provide  some 
helpful  feedback  in  planning  for  future  activities  of  (the  initiative). 

If  you  have  any  questions,  please  feel  free  to  call  us.  We  can  also  further  discuss 
the  survey  findings  at  our  next  meeting. 


cc:  Program  Officer,  Funding  Source 
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Assessing  the  Importance  of  Outcomes 


Not  all  changes  made  by  the  initiative  are  equally  important.  The  survey  of 
outcomes  asks  those  involved  in  the  initiative,  funding  agents,  and  outside 
experts  to  rate  the  significance  of  changes  that  the  initiative  has  made  to  reduce 
the  risks  of  CVD.  Consistent  with  the  evaluation  framework  outlined  in  Chapter 
3,  it  helps  assess  intermediate  outcomes  for  the  initiative.  A  key  evaluation 
question  is  addressed  by  this  survey:   "Were  the  community  changes 
important  to  reducing  risks  for  CVD?" 


Survey  of  outcomes:     What  and  why? 


What  is  it? 


•  A  way  to  assess  the  significance  of  the  community  changes  facilitated  by  the 
partnership. 

Why  do  it? 

•  To  help  identify  the  more  important  accomplishments  of  the  initiative. 

•  To  direct  attention  to  more  valued  changes  to  be  sought. 


Both  members  of  the  partnership  and  outside  experts  can  assess  the  importance 
of  accomplishments.  Outside  experts  may  include  physicians,  epidemiologists, 
national  activists  in  physical  activity,  tobacco  control,  or  other  risk  factors,  and 
experts  in  designing  programs  to  prevent  CVD. 

The  survey  asks  respondents  to  rate  the  contribution  of  all  the  changes 
individually  and,  taken  together,  to  reducing  risks  for  CVD.   Leaders  can  use 
these  data  to  adjust  the  focus  of  efforts  to  areas  that  are  valued  by  people 
involved  and  to  areas  that  contribute  most  substantially  to  reducing  risks  for 
CVD. 
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This  chapter  outlines  how  to  use  a  survey  of  outcomes  to  assess  the  importance 
of  community  changes  facilitated  by  the  initiative. 


Survey  of  outcomes:     How  to's 

•  Develop  a  survey  that  lists  all  community  changes  (accomplishments) 
facilitated  by  the  initiative. 

•  Conduct  the  survey  with: 

Members  (with  experiential  knowledge  of  the  partnership) 

Funders 

Outside  experts  (with  specialized  knowledge  in  community  efforts 
to  prevent  CVD). 

•  Obtain  average  ratings  for  each  change  on  its  significance  to  reducing  risks 
for  CVD. 

•  Use  the  survey  data  to  redirect  efforts  of  the  initiative. 


Evaluators,  or  persons  responsible  for  conducting  the  survey,  design  the  survey 
from  a  list  of  the  initiative's  accomplishments  provided  by  staff  or  active 
members.  The  list  of  accomplishments  can  be  obtained  by  using  community 
changes  identified  with  the  monitoring  system  (see  Chapter  4).  The  survey 
items  should  acknowledge  groups  and  people  who  helped  make  the  changes 
happen.  Project  staff  develop  a  list  of  people  to  receive  the  survey.  The  survey 
is  distributed  by  mail  or  during  coalition  meetings.   Respondents  return 
completed  questionnaires  to  the  evaluation  team,  who  then  summarizes  the 
data. 

The  entire  process  (from  developing  the  survey  to  final  summarizing  of  the  data) 
will  take  about  two  to  three  months.  Staff  of  the  initiative  spend  approximately 
10  hours  in  collaborating  on  developing  the  survey  and  distributing 
questionnaires.  We  recommend  that  the  outcome  survey  be  conducted  every 
several  years  and/or  in  the  last  year  of  the  grant. 

Materials  at  the  end  of  this  chapter  include  a  generic  outcome  survey  that  may 
be  used  as  a  format  for  designing  the  survey,  a  sample  cover  letter,  and  a  sample 
memorandum  of  results. 
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Adjusting  the  System  to  Meet  Your  Needs 


When  deciding  how  to  assess  the  importance 
of  outcomes,  consider: 

•  What  members  and  leaders  of  the  community  partnership  want  to  know. 

•  Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

•  Resources  available  to  address  these  questions. 


For  community  partnerships  with  HIGH  resources  and  requirements  for 
evaluation,  consider  doing  annual  assessments  of  the  significance  of  community 
changes  facilitated  by  the  partnership.  This  might  presume  the  availability  of 
coalition  staff  to  print  and  distribute  approximately  100  questionnaires;  a 
person  with  basic  skills  in  data  entry  and  analysis;  and  a  modest  budget  for 
printing,  distribution,  data  analysis,  and  other  related  expenses. 

For  community  partnerships  with  LOW  resources  and  requirements,  consider 
assessing  the  importance  of  community  changes  informally  or  less  often. 
Recommended  strategies  for  assessing  the  importance  of  outcomes  on  a 
shoestring  budget  are  listed  below. 

Recommended  Strategies  for  Assessing  the  Importance  of 
Outcomes  on  a  Shoestring  Budget 

•  Distribute  the  survey  only  to  coalition  members,  and  do  so  at  coalition 
meetings.  If  resources  permit,  consider  sending  questionnaires  to  the  few 
constituents  the  coalition  would  most  like  to  hear  from.  Summarize  the  data 
on  perceived  contributions  of  the  community  changes  to  reducing  the  risks 
forCVD. 

•  Ask  persons  from  local  or  State  health  departments  who  are  not  involved 
with  the  initiative  to  serve  as  experts  in  assessing  the  importance  of 
outcomes. 

•  Instead  of  using  a  survey,  collect  information  in  informal  listening  sessions. 
Flip  charts,  a  writing  board  or  chalkboard,  or  plain  paper  can  be  used  to 
capture  member  input  on: 

More  significant  changes  in  programs,  policies,  and  practices 

Less  significant  changes 

Adjustments  in  goals  to  improve  outcomes. 

•  Local  citizen  organizations  might  facilitate  listening  sessions  free  of  charge. 
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Materials  for  Assessing  the  Importance  of  Outcomes 

The  following  section  provides  a  sample  survey  of  outcomes  and  other 
background  information.   Materials  that  can  be  tailored  to  suit  any  CVD 
prevention  effort  that  plans  to  assess  the  importance  of  its  accomplishments  are 
included. 


Materials  include  the  following 

A  sample  cover  letter  for  an  outcome  survey  (Figure  6.1)  that  is  sent  out 
with  the  survey  and  describes  the  purpose  of  the  survey,  how  the  survey 
should  be  filled  out,  and  what  will  be  done  with  the  results. 

A  format  for  a  generic  survey  (Figure  6.2)  outlining  the  components  and 
suggested  format  that  can  be  used  as  a  guide  when  developing  your  Survey 
of  Outcomes. 

An  example  memorandum  (Figure  6.3)  that  demonstrates  how  results  of  a 
Survey  of  Outcomes  can  be  summarized  and  presented.  Typically, 
community  partnerships  use  this  or  some  similarly  written  document  to 
inform  interested  parties  of  the  survey  results. 
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Figure  6.1 
Sample  cover  letter  for  the  Outcome  Survey 


Date 


Dear  Members  of  (the  initiative): 

The  purpose  of  the  attached  outcome  survey  is  to  get  your  feedback  on  the 
accomplishments  of  (the  initiative).  As  you  know,  the  mission  of  (the  initiative) 
is  to  reduce  the  risks  of  heart  disease  in  (city/county/state) .  This  survey  lists 
each  change  in  program,  policy,  or  practice  facilitated  by  (the  initiative).   Note 
that  many  of  these  changes  were  facilitated  in  collaboration  with  partners  of  (the 
initiative). 

Please  complete  each  question  by  circling  the  number  that  illustrates  how 
important  each  community  change  is  to  the  mission  of  reducing  risks  for  CVD. 

Once  we  receive  all  the  completed  questionnaires,  we  will  analyze  and 
summarize  the  data.  The  results  should  help  identify  the  accomplishments  of 
the  most  and  least  importance  from  the  perspective  of  coalition  members  and 
outside  experts  in  prevention  of  CVD. 

To  protect  anonymity,  use  the  enclosed  self-addressed  envelope  and  return  your 
completed  survey  to  our  e valuators,  (name  them). 

Thanks  in  advance  for  your  valuable  time  and  feedback. 


Best  regards, 


Executive  Director 


Enclosure 
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Figure  6.2 
Generic  Outcome  Survey 


(return  address) 


(due  date) 


PLEASE  RETURN  TO  _ 
PLEASE  RETURN  BY  _ 

(Date) 

Staff  and  members  of  (the  initiative)  have  been  involved  in  efforts  that  resulted  in 
a  variety  of  community  changes  related  to  the  mission  of  preventing  CVD. 

This  survey  lists  XX  community  changes  that  were  facilitated  or  supported  by 
the  initiative's  efforts.  For  each  survey  item,  please  circle  the  number  that  best 
describes  how  important  each  community  change  is  to  the  initiative's  mission  of 
reducing  risks  for  CVD.  Use  the  following  scale  to  rate  your  answers. 


Very  Unimportant  Neither 

unimportant  unimportant 

nor  important 


4 

Important 


Very 
important 


IMPORTANCE 

to  the  mission  of  reducing  risks 

of  cardiovascular  diseases 


Community 
Changes 

Very 
unimportant 

1)  ( 

(Date) 

) 

1 

2)  ( 

(Date) 
_) 

1 

Comments: 

Very 
important 


Taken  together,  how  important 
are  the  above  community  changes 
in  accomplishing  the  mission  of 
reducing  risks  for  CVD? 
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Figure  6.3 

Sample  memorandum  of  results  for  the  Outcome  Survey 

(adapted  from  the  Kansas  LEAN  Survey) 


Date: 


To:         (Leadership  and  Board  of  Directors) 

From:     The  Evaluation  Team 

Re:         Results  of  the  Outcome  Survey  for  (the  initiative). 

Enclosed  please  find  the  results  of  the  completed  Outcome  Survey  for  (the 
initiative).  The  purpose  of  the  survey  was  to  assess  the  importance  of 
community  changes  resulting  from  (the  initiative's)  efforts.  Surveys  were 
distributed  to  96  initiative  or  Task  Force  members  and  5  experts  in  the  areas  of 
CVD  or  initiative  development.  The  overall  response  rate  was  very  high  (47%). 
Nearly  one-half  (49%)  of  Initiative  and  Task  Force  Members  returned  their 
surveys.  The  attached  report  includes  the  average  ratings  and  the  range  of 
responses  for  each  question  listed  in  order  of  occurrence  on  the  survey. 
Respondents'  written  comments  are  listed  verbatim. 

In  general,  respondents  rated  the  outcomes  facilitated  by  (the  initiative)  as 
important  in  accomplishing  the  mission  of  preventing  CVD.  The  highest  mean 
importance  rating  was  4.4  (School  Intervention  Projects  in  Salina  and  Dighton). 
The  lowest  mean  importance  rating  was  3.4  (Home  Economics  Sandwich 
contest).  Taken  together,  the  contribution  of  the  reported  community  changes 
to  the  mission  was  rated  between  important  and  very  important  (mean  =  4.4). 

Many  of  the  written  comments  affirmed  the  high  importance  ratings. 
Comments  were  provided  on  the  methods  used  to  educate  others  (e.g., 
"Excellent  approaches  to  educating  educators  as  well  as  individuals  regarding 
reduction  of  fat")  and  competence  of  staff  members  (e.g.,  "Staff  has  done  a 
tremendous  job"). 

Respondents  indicated  that  community  changes  produced  by  (the  initiative)  are 
important  to  the  mission  of  preventing  CVD.  The  listing  of  the  many 
accomplishment  of  (the  initiative)  and  the  ratings  of  their  importance  are  cause 
for  celebration!  You  should  be  proud  of  these  accomplishments.  If  you  have  any 
questions,  please  feel  free  to  call  us. 


cc:  Program  Officer,  Funding  Source 
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Using  Behavioral  Data  and  Surveys 


It  is  important  to  document  the  partnership's  impact  on  distal  outcomes  such  as 
reported  behaviors  associated  with  risks  for  CVD.  Behavioral  surveys  are  a 
strategy  for  measuring  reported  changes  in  the  behaviors  of  individuals. 
Behavioral  surveys  help  answer  the  following  key  evaluation  question:  '7s  there 
a  change  in  behavior  related  to  risks  for  CVD?" 


Behavioral  surveys:    What  and  why? 

What  are  they? 

•  A  way  to  find  out  about  everyday  behaviors  or  habits  that  increase  local 
citizen's  risks  for  developing  CVD. 

Why  obtain  this  information? 

•  Everyday  behaviors  are  associated  with  more  than  50%  of  all  CVD 
deaths — it  is  important  to  identify  what  behaviors  local  citizens  need 
to  change. 

•  To  provide  data  for  program  evaluation. 

•  To  help  identify  the  level  of  risks  for  CVD  in  a  community  or  state. 
Why  be  careful? 

•  Behavioral  surveys  are  costly. 

•  They  may  be  insensitive  to  change. 

•  They  focus  on  the  individual  and  not  on  the  environment. 
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This  chapter  describes  behavioral  surveys  and  how  to  obtain  and  use  data  from 
them.  Everyday  behaviors  or  habits  that  are  associated  with  health  problems  are 
called  behavioral  risk  factors.  Behavioral  surveys  include  questions  about 
behaviors  that  put  people  at  risk  for  CVD  and  other  diseases.  These  CVD  risk 
factors  include  inadequate  physical  activity,  poor  diet,  and  smoking.  It  is 
estimated  that  these  activities  are  responsible  for  more  than  half  of  the 
cardiovascular  diseases  in  the  United  States.1 

Conducting  a  behavioral  survey  of  risk  factors  for  CVD  provides  valuable 
information  about  the  health  status  of  a  community  or  State.  A  community  can 
use  these  data  to  determine  what  behaviors  need  to  change  and  to  identify 
certain  groups  who  are  at  particular  risk  for  CVD.  Furthermore,  behavioral  data 
can  provide  important  information  for  deciding  what  the  project  should  work  on 
first,  planning  and  implementing  community  efforts,  and  tracking  changes  in 
local  citizens. 

Behavioral  data  strengthen  the  evaluation  of  community  partnerships.  Most  of 
the  initiative's  effort  is  directed  toward  creating  local  programs  and  changing 
policies.  These  community  changes  should,  in  turn,  help  change  citizens'  health 
behaviors.  Behavioral  surveys  provide  information  on  how  citizens'  behaviors 
change  in  response  to  the  efforts  of  the  partnership. 

The  success  of  a  change  in  program,  policy,  or  other  aspect  of  the  environment 
thought  to  reduce  risks  for  CVD  is  validated  by  associated  changes  in  behavior. 
Surveillance  of  behavioral  risk  factors  provides  a  basis  for  both  developing  and 
evaluating  programs  to  prevent  CVD. 


Cautions  in  Using  Behavioral  Surveys 

Although  behavioral  surveys  can  be  excellent  tools  for  planning  and  for 
monitoring  program  effects,  several  issues  and  limitations  should  be  considered. 
First,  because  many  behaviors  are  "caused"  by  a  variety  of  factors,  changes  in  a 
particular  behavior  may  or  may  not  be  due  to  the  efforts  of  the  partnership. 
Second,  it  might  also  be  necessary  to  sample  a  large  number  of  people  in  order 
to  know  if  the  initiative  is  having  an  effect.  Many  of  the  people  in  that  sample 
may  not  have  been  exposed  to  the  intervention,  may  not  be  at  risk  in  the  first 
place  (e.g.,  nonsmokers  or  those  already  physically  active),  or  may  have 
changed  their  CVD  risk  behaviors  for  other  reasons. 

Third,  monitoring  change  in  behavior  and  attributing  that  change  to  your  efforts 
is  much  more  difficult  than  using  the  data  to  guide  program  development. 
Fourth,  even  with  rather  large  samples,  it  may  also  be  inappropriate  or  difficult 
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to  look  for  change  in  behaviors  for  particular  groups  such  as  minorities  or 
specific  age  groups.  Fifth,  it  may  be  inappropriate  to  generalize  findings  from 
one  community  or  region  to  another.  Finally,  behavior  surveys  also  tend  to 
"blame  the  victim."  They  may  focus  explanations  of  the  cause  of  behavior  on 
the  individual  rather  than  the  environmental  conditions  that  influence  behavior. 
In  short,  behavioral  surveys  can  be  costly,  complex,  insensitive,  or  even 
inappropriate. 

Several  strategies  help  reduce  these  problems  with  behavioral  surveys.  First, 
consider  purposely  sampling  only  those  exposed  to  the  initiative.  Second, 
include  questions  on  availability  and  opportunity  for  behaviors.  (For  example, 
have  you  noticed  any  change  in  the  availability  of  low  fat  foods  in  the  cafeteria? 
How  often  are  you  in  a  location  where  you  could  take  a  walk  or  go  jogging?  If 
you  play  tennis,  how  often  do  you  have  to  wait  for  a  court?)  Finally,  use  other 
indicators  of  program  success  to  complement  the  behavioral  survey. 


Using  Behavioral  Surveys 

CDC  has  developed  the  Behavioral  Risk  Factor  Surveillance  System  (BRFSS)  for 
adults.  It  is  now  used  by  all  States  to  help  track  the  prevalence  of  behavioral  risk 
factors  in  the  United  States.  This  surveillance  system  uses  a  standardized 
protocol  to  interview  randomly  selected  adults. 

The  BRFSS  is  made  up  of  a  core  set  of  questions  and  has  several  optional 
modules  related  to  CVD.  Questions  are  available  on  behaviors  to  control  or 
monitor  hypertension,  cholesterol,  and  diabetes;  amount  of  leisure  time  physical 
activity;  typical  diet;  efforts  to  modify  one's  diet;  and  weight  control  practices. 
The  box  below  lists  sample  questions  from  the  BRFSS. 
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Sample  questions  from  the 
1995  Behavioral  Risk  Factor  Questionnaire 

On  the  average,  about  how  many  cigarettes  a  day  do  you  now  smoke? 

Have  you  smoked  at  least  100  cigarettes  in  your  entire  life? 

Are  you  eating  either  fewer  calories  or  less  fat  to  lose  weight  or  to  keep 
from  gaining  weight? 

Not  counting  carrots,  potatoes,  or  salad,  how  many  servings  of  vegetables 
do  you  usually  eat? 

During  the  past  month,  did  you  participate  in  any  physical  activities  or 
exercises  such  as  running,  calisthenics,  golf,  gardening,  or  walking  for 
exercise? 

How  many  times  per  week  or  per  month  did  you  take  part  in  this  activity 
during  the  past  month? 

In  the  past  12  months,  has  a  doctor,  nurse,  or  other  health  professional 
given  you  advice  about  your  weight  (to  lose,  gain,  or  maintain)? 

Have  you  ever  been  told  by  a  doctor  or  other  health  professional  that  your 
blood  cholesterol  is  high? 

Have  you  ever  been  told  by  a  doctor,  nurse,  or  other  health  professional 
that  you  have  high  blood  pressure? 

Have  you  ever  been  told  by  a  doctor  that  you  have  diabetes? 


The  CDC  has  also  developed  a  Youth  Risk  Behavior  Survey  (YRBS)  to  assess 
risk  behaviors  among  youth.  The  YRBS  is  typically  conducted  in  schools  but 
also  has  modules  for  "out  of  school  youth."  CVD  risk  behaviors,  such  as 
smoking,  begin  in  youth  and  continue  into  adulthood.  Community  efforts  of 
primary  prevention  of  CVD  might  use  both  the  BRFSS  and  YRBS  to  monitor 
healthy  habits  or  behaviors  among  adults  and  youth. 
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Adjusting  the  System  to  Meet  Your  Needs 


When  deciding  what,  if  any,  behavioral  measures 
to  collect,  consider: 

•  What  members  and  leaders  of  the  community  partnership  want  to  know. 

•  Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

•  Resources  available  to  conduct  the  survey. 

•  What  data  are  already  available. 


For  community  partnerships  with  HIGH  resources  and  requirements  for 
evaluation,  consider  collecting  all  of  these  possible  behavioral  measures  and 
others  of  interest.  Although  resource  needs  could  vary  with  the  sample  size  and 
population,  conducting  behavioral  surveys  is  fairly  expensive.  The  State  BRFSS 
group  or  a  university  survey  team  can  provide  an  estimate  of  budget  and  time 
requirements. 

For  community  partnerships  with  LOW  resources  and  requirements,  consider 
not  collecting  any  behavioral  data  or  doing  so  with  a  small  sample  of  people. 
This  might  presume  little  or  no  evaluation  budget  and  virtually  no  reporting 
requirements.  Staff  might  review  data  available  from  other  sources.  This 
presumes  staff  are  available  for  several  hours  a  week  or  less,  with  minimal 
training  in  evaluation. 
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Recommended  Strategies  for  Collecting  Behavioral  Measures 

on  a  Shoestring  Budget 

Use  a  smaller  sample  of  survey  participants,  accepting  lower  confidence 
levels  for  the  resulting  behavioral  data. 

Collect  behavioral  survey  data  less  often,  perhaps  only  twice  (once  at  the 
beginning  and  again  at  the  end  of  the  project)  after  several  years  of 
intervention  activities. 

Piggyback  behavioral  questions  related  to  CVD  on  other  existing  surveys; 
for  example,  school  surveys  (e.g.,  the  YRBS)  or  statewide  behavioral  risk 
surveys  (BRFSS)  sponsored  by  the  State  health  department. 

Collaborate  with  local  colleges  or  universities  to  conduct  surveys  as  part  of 
a  class  project. 

Instead  of  conducting  behavioral  surveys,  rely  on  archival  measures  of  the 
products  of  behavior  (e.g.,  tobacco  consumption  from  excise  tax  data)  or 
measure  changes  in  the  conditions  known  to  prompt  or  inhibit  the 
behaviors  such  as  access  to  tobacco  products  or  smoking  ordinances. 


Background  Information  and  Materials  for 
Using  the  BRFSS 

This  section  provides  background  information  on  the  BRFSS,  including  a  list  of 
optional  modules  and  addresses  of  people  to  contact.  A  copy  of  relevant 
portions  of  the  BRFSS  questionnaire  is  provided  in  Appendix  C.  Each  State  has 
a  BRFSS  coordinator,  and  statewide  data  are  available  on  an  annual  basis.  Call 
the  local  coordinator  and  let  them  know  your  State  or  community  is  interested 
in  conducting  a  behavioral  survey. 
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Behavioral  surveys:  How  to  obtain  and  use  the  data 

•  Obtain  existing  behavioral  data  from  local  agencies  (if  necessary,  select  and 
conduct  behavioral  surveys  in  collaboration  with  local  and  State  agencies  or 
universities). 

•  Plot  the  data  and  provide  feedback  on  CVD  risks  factors,  such  as  smoking, 
fat  intake,  or  exercise  levels. 

•  Use  risk  factor  data  to  increase  awareness  and  to  plan  for  programs  and 
public  concern  for  CVD,  and  evaluate  the  effects  of  programs. 

•  It  is  also  recommended  to  use  a  graphics  program  to  graph  the  data. 


Behavioral  surveys  should  be  culturally  appropriate  and  should  collect 
appropriate  data  for  program  evaluation.  For  instance,  local  programs  modified 
the  BRFSS  questionnaire  and  conducted  face-to-face  interviews  for  some  Native 
American  reservations  and  rural  Hispanic  communities  where  telephones  were 
not  commonly  available.  Other  programs  have  revised  the  questions  or  made 
adjustments  to  fit  cultural  patterns  and  norms  such  as  matching  the  gender  of 
interviewers  to  the  person  being  interviewed  for  one  Native  American  tribe, 
training  community  members  as  interviewers,  and  eliminating  or  revising 
sensitive  questions.  Questions  have  been  translated  into  other  languages, 
including  Spanish  and  Chinese,  and  surveys  have  been  revised  to  collect 
information  on  behaviors  not  usually  monitored  by  the  statewide  BRFSS. 

The  BRFSS  questions  are  considered  a  valid  measure  of  the  State's  level  of  risks 
for  CVD  and  other  health  problems.  Because  it  is  costly,  consider  the  cautions 
noted  above  when  determining  whether  to  obtain  behavioral  data. 
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Using  Data  from  Behavioral  Surveys 

Four  figures  are  provided  to  help  illustrate  how  the  data  collected  from  a 
behavioral  survey  may  be  used  and  shared.  Figure  7.1  illustrates  the  prevalence 
of  hypertension  in  an  average  U.S.  county  for  1986  and  1991  and  the  State 
prevalence  for  1989-1990.  At  the  county  level,  the  percentage  of 
hypertension  for  males  decreased  slightly  from  1986  to  1991.  For  females,  the 
percentage  of  hypertension  decreased  markedly  from  1986  to  1991.  The  total 
population  for  the  county  also  showed  a  decrease  in  the  prevalence  of 
hypertension.  Data  from  the  State  show  that  the  levels  of  hypertension  for 
males  and  females  are  lower  than  the  county  level.  These  data  could  be  shared 
with  local  officials  to  increase  awareness  about  the  problem  and  to  secure 
community  involvement. 


Figure  7.1 
Prevalence  of  hypertension  in  one  U.S.  county,  1986  vs.  1991 
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Figure  7.2  shows  the  percentage  of  survey  respondents  who  reported  cigarette 
smoking  in  one  U.S.  county  and  the  State.  At  the  county  level,  the  percentage 
of  cigarette  smoking  for  males,  females,  and  the  total  population  increased 
slightly  from  1986  to  1991.  Reported  smoking  at  the  county  and  State  levels 
were  comparable.  These  data  can  be  used  to  establish  the  level  of  the  problem 
at  the  State  and  local  level. 


Figure  7.2 
Prevalence  of  cigarette  smoking  in  one  U.S.  county,  1986  vs.  1991 
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Figure  7.3  shows  the  prevalence  of  adults  in  California  who  reported  being 
overweight  from  1984  to  1993.  The  data  have  been  age  adjusted  using  the 
1990  California  census  population.  The  data  show  a  gradual  increase  in  the 
prevalence  of  overweight  adults  since  1984.  They  could  be  used  to  suggest  the 
need  for  creating  new  programs,  practices,  or  policies  to  reduce  overweight 
among  adults. 

Figure  7.4  shows  the  percentage  of  adults  with  no  leisure  time  for  exercise. 
Forty-one  percent  of  Black  females,  36%  of  Hispanic  females,  and  29%  of 
Hispanic  males  reported  not  exercising  during  leisure  time.  These  data  can  be 
used  to  suggest  that  special  exercise  programs  or  messages  may  need  to  be 
developed  for  particular  groups  such  as  Black  females,  Hispanic  females,  and 
Hispanic  males. 

Figure  7.3 
Prevalence  of  overweight  adults  in  California,  1984-1993 
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Source:  California  BRFSS,  California  Department  of  Health  Services,  Cancer  Surveillance  Sec- 
tion. 
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Figure  7.4 

Prevalence  of  adults  with  no  leisure-time  physical  activity, 

by  race  and  gender,  1 993 
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Source:  California  BRFSS,  California  Department  of  Health  Services,  Cancer  Surveillance 
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Learning  More  About  the  BRFSS 

To  learn  more  about  the  BRFSS  questionnaire  and  how  to  obtain  data,  first 
contact  the  BRFSS  coordinator  in  the  State  health  department.  A  copy  of  the 
BRFSS  and  the  modules  can  be  obtained  from  the  CDC  through  the  Office  of 
Surveillance  and  Analysis,  Behavioral  Risk  Factor  Surveillance  Branch,  4770 
Buford  Highway,  Mailstop  K-30,  Atlanta,  Georgia  30341-3724.  CDC  analyzes 
the  data  that  are  collected  from  administering  the  statewide  BRFSS. 
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Core  questions  of  the  BRFSS,  and  optional  modules  are  listed  in  Table  7.1. 
Those  of  particular  relevance  to  community  partnerships  to  prevent  CVD  are 
noted  with  an  asterisk  and  are  included  in  Appendix  C. 


Table  7.1 
BRFSS  core  sections  and  optional  modules 


Core  Sections 

Optional  Modules 

Section  1: 

Health  Status 

Module  1: 

Smokeless  Tobacco  Use 

Section  2: 

Health  Care  Access 

Module  2: 

Fruits  and  Vegetables* 

Section  3: 

Hypertension  Awareness* 

Module  3: 

Diabetes 

Section  4: 

Cholesterol  Awareness* 

Module  4: 

Exercise* 

Section  5: 

Diabetes 

Module  5: 

Weight  Control* 

Section  6: 

Injury  Control 

Module  6: 

Years  of  Healthy  Life 

Section  7: 

Tobacco  Use* 

Module  7: 

Quality  of  Life 

Section  8: 

Alcohol  Consumption 

Module  8: 

Health  Care  Utilization 

Section  9: 

Demographics 

Module  9: 

Oral  Health 

Section  10: 

Women's  Health 

Module  10: 

Firearms 

Section  11: 

Immunization 

Section  12: 

Colorectal  Cancer 

Section  13: 

AIDS  Knowledge  and  Testing 
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Community-level  Indicators  of 
Cardiovascular  Health 


Community-level  indicators  of  CVD  provide  a  measure  of  whether  the 
community  partnership  successfully  improved  cardiovascular  health  in  the 
community.  In  the  framework  for  health  promotion  presented  in  Chapter  3, 
reduced  risks  for  CVD  may  be  estimated  by  monitoring  distal  outcomes.  This  is 
because  a  project  must  plan  and  implement  efforts  for  several  years  and  create 
lots  of  community  changes  before  it  can  have  an  effect  on  the  community-wide 
level  of  CVD  or  its  risk  factors.  Two  core  evaluation  questions  are  addressed 
using  community-level  indicators:  "Does  the  initiative  have  a  community-level 
outcome  related  to  risks  for  CVD?"  and  '7s  community-level  outcome 
related  to  changes  facilitated  by  the  initiative?" 


Community  indicators:    What  and  why? 
What  are  they? 

•  They  are  a  way  to  obtain  information  about  the  more  distal  effects  of  the 
CVD  prevention  effort. 

•  They  provide  an  indicator  of  cardiovascular  health  of  the  community, 
similar  to  the  way  "economic  indicators"  (such  as  Median  Family  Income) 
measure  the  fiscal  health  of  the  United  States. 

•  Some  indicators  measure  a  community's  "health  potential"  by  counting 
opportunities  in  the  community  for  low-fat  eating,  physical  activity,  and 
smoke-free  living  (e.g.,  number  of  physical  activity  facilities  per  capita). 

•  Other  indicators  measure  community  levels  of  major  risk  factors  for  CVD 
(e.g.,  smoking,  obesity,  and  sedentariness). 
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Community  indicators:    What  and  why?  (Continued) 
Why  do  it? 

•  To  see  if  a  CVD  initiative  is  improving  the  cardiovascular  health  of  the 
community. 

•  To  see  if  special  projects  of  a  CVD  prevention  effort  are  reducing  risks  for 
CVD. 

•  To  get  a  better  picture  of  the  initiative's  effects  on  your  community  as  a 
whole. 


The  incidence  and  prevalence  of  CVD  may  appear  to  be  the  ideal  measure  for 
program  effects — after  all,  the  ultimate  outcome  of  the  initiative  is  to  reduce 
CVD.  But,  it  may  take  decades  for  changes  in  CVD  risk  factors  to  result  in 
changes  in  rates  of  CVD.  Projects  simply  cannot  wait  that  long  to  see  if  they  are 
effective.  Cardiovascular  health  indicators  are  a  quicker  way  to  examine 
whether  the  partnership  has  made  the  community  a  more  health-promoting 
place.  Indicators  do  so  by  measuring  risk  factors  for  CVD  as  well  as  community- 
based  opportunities  for  developing  lifelong,  heart-healthy  habits.  Table  8.1 
provides  examples  of  indicators  that  measure  the  cardiovascular  health  of 
communities. 

Table  8.1 
Examples  of  community-level  outcome  indicators 


Tobacco  Control 


Physical  Activity 


Diet 


Per  capita 
consumption  of 
cigarettes 


Percentage  of 
worksites  with  clean 
indoor  air  policies 

Prevalence  of 
tobacco  use 


Number  of  physical  activity  facilities 
(fields,  gyms,  pools,  etc.)  per  capita 
in  schools 


Miles  of  walking  trails  per  capita 


Prevalence  of  physical  activity 


Presence  or  absence  of 
low-fat  food  alternatives 
in  vending  machines  in 
public  buildings 

Proportion  of  school 
menu  items  that  meet 
nutritional  guidelines 

Prevalence  of 
overweight 


See  nutrition  objective  2.17  in  U.S.  Department  of  Health  and  Human  Services.  (1990). 
Healthy  people  2000:  National  health  promotion  and  disease  prevention  objectives. 
(DHHS  Publication  No.  PHS  91-50213).  Washington,  DC:  U.S.  Government  Printing  Office. 

Note:  These  indicators  can  be  used  as  community-level  outcomes  although  they  reflect  individ- 
ual data  aggregated  for  a  community. 
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Measuring  the  health  potential  of  a  community  represents  a  shift  in  focus  for 
community  health  promotion  efforts,  and  it  represents  the  belief  that  a  health- 
promoting  community  will  foster  healthier  citizens.  Community-level  indicators 
of  CVD  attempt  to  measure  just  how  health-promoting  a  community  is. 
Candidate  community-level  indicators  of  cardiovascular  health  have  been 
identified  by  experts  in  CVD  prevention  and  evaluation.  Lists  of  these  candidate 
indicators,  categorized  by  the  main  CVD  risk  factors  of  tobacco,  physical 
inactivity,  and  diet,  are  presented  in  Tables  8.4,  8.5,  and  8.6  at  the  end  of  this 
chapter.  These  indicators  are  being  reviewed  for  quality  (accuracy  and 
sensitivity)  and  feasibility  (availability  and  cost).  The  accuracy  and  feasibility  of 
these  indicators  need  to  be  validated  in  communities.   In  the  meantime,  CVD 
projects  might  collect  a  few  indicators  that  best  correspond  to  their  project's 
efforts.  Many  of  the  indicators  found  in  Tables  8.4,  8.5,  and  8.6  are  also 
community  changes  (see  Chapter  4).  Figure  10.4  in  Chapter  10  illustrates  how 
data  from  community-level  indicators  can  be  graphed  and  displayed. 


Community-level  indicators:   How  to's 

•  Select  which  indicators  the  initiative  will  collect. 

•  Collect  the  indicators. 

•  Summarize  and  graph  the  data. 

•  Present,  at  least  annually,  the  graphs  of  community-level  indicators  to 
community  leaders,  project  funders,  and  other  interested  individuals  or 
groups. 

•  Present  the  data  as  needed  to  keep  prevention  of  CVD  on  the  local  public 
agenda. 


It  may  be  difficult  to  determine  the  best  indicators  for  evaluating  project 
efforts — it  may  be  useful  to  form  a  temporary  task  force  for  selecting  indicators. 
Members  of  the  task  force  and  project  staff  can  oversee  selecting,  collecting, 
and  disseminating  data  on  community-level  indicators.  The  task  force  should 
select  indicators  and  collect  data  on  their  chosen  indicators  within  the  first  year 
of  project  initiation.  For  more  information  on  community-level  indicators  of 
CVD,  see  the  section  on  background  information  and  materials  at  the  end  of 
this  chapter. 
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Adjusting  the  System  to  Meet  Your  Needs 

For  projects  with  HIGH  resources  and  requirements  for  evaluation,  consider 
collecting  several  cardiovascular  health  indicators.  To  do  so,  the  program  must 
commit  staff  time  in  order  to  identify  sources  of  information.  The  project  should 
have  access  to  someone  with  basic  skills  in  evaluation.  In  addition  to  these 
personnel  costs,  selecting,  collecting,  and  graphing  data  will  require  a  modest 
budget  for  supplies. 

For  community  programs  with  LOW  resources  and  requirements,  consider  not 
collecting  any  cardiovascular  indicators  or  only  a  few.  If  less  data  are  collected, 
programs  may  choose  to  present  data  in  a  table  format  rather  than  needing  to 
graph  data.  Recommended  strategies  for  collecting  cardiovascular  health 
indicators  on  a  shoestring  budget  are  listed  below. 


Recommended  Strategies  for  Collecting  Cardiovascular 
Health  Indicators  on  a  Shoestring  Budget 

See  if  any  project  partners  have  already  collected  these  data  or  if  any  other 
group  has  an  interest  in  doing  so  (e.g.,  State  or  county  health  departments, 
State  or  county  Parks  and  Recreation  departments,  businesses  in  the  private 
sector,  or  universities). 

Collect  data  on  only  a  few  indicators. 

Collect  data  less  often,  perhaps  only  twice  (once  at  the  beginning  and  again 
at  the  end  of  the  project). 


Background  Information  and  Materials  for 
Community-level  Indicators 

The  previous  section  described  community-level  indicators  of  cardiovascular 
health  and  their  importance  in  examining  the  effects  of  community-based 
initiatives  for  prevention  of  CVD.  This  section  is  written  for  individuals  who  are 
responsible  for  selecting  indicators  for  programs.  Subsections  include 
background  information  on  criteria  to  use  in  selecting  indicators;  the  selection 
process;  and  tips  on  how  to  collect,  graph,  and  present  the  data.  The  chapter 
concludes  with  a  menu  of  potential  community-level  indicators  (see  Tables  8.4, 
8.5,  and  8.6). 
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Selecting  Community-level  Indicators 

A  health-promoting  community  creates  healthier  citizens.  Community-level 
indicators  of  CVD  attempt  to  measure  the  degree  to  which  a  community  is 
promoting  health.  All  measurement  systems  have  to  meet  certain  criteria  in 
order  to  be  deemed  appropriate  measures  of  reality.  For  example,  a  scale  has  to 
meet  certain  standards  of  accuracy,  sensitivity,  and  reliability  to  be  considered  a 
good  measure  of  an  object's  weight.  Because  each  community  is  distinct  and 
each  CVD  prevention  effort  is  different,  CVD  initiatives  must  take  an  active  role 
in  determining  which  indicators  are  the  best  measures  of  their  programs'  efforts. 
Those  responsible  for  selecting  measures  should  use  several  criteria  for 
determining  which  indicators  to  collect. 

Community-level  indicators  must  be  of  sufficient  quality,  feasible  to  collect,  and 
reliable  and  valid  to  be  considered  a  good  measure  of  program  effects.  Each 
attribute  is  considered  below. 

Quality.  The  quality  of  data  for  an  indicator  is  determined  by  its  accuracy  and 
sensitivity. 

Accuracy.  This  criterion  encourages  those  responsible  for  the  evaluation  to 
examine  how  carefully  and  consistently  data  are  collected,  recorded,  and  stored. 
Sometimes  there  are  so  many  flaws  that  the  data  are  inaccurate. 

Sensitivity.  Those  responsible  for  the  evaluation  must  also  consider 
whether  each  indicator  measures  the  effects  of  the  initiative.  An  indicator 
should  correspond  to  the  types  of  changes  the  program  hopes  to  make  in  the 
community.  Also,  the  measure  should  be  sensitive  enough  to  register  when 
changes  occur.  For  example,  if  a  weight  reduction  program  creates  changes  in 
one  town,  county-wide  data  on  levels  of  overweight  may  not  register  reductions 
in  overweight  that  are  due  to  program  efforts.  The  county  may  contain  many 
more  times  the  population  of  the  intervention  town,  so  reductions  in  the  levels 
of  overweight  in  the  town  will  not  be  detectable. 

Feasibility.  The  feasibility  of  an  indicator  is  determined  by  whether  the  data  are 
already  available,  or  previously  collected  for  the  intervention  site,  and  the  costs 
(in  both  effort  and  money)  of  collecting  or  analyzing  the  data. 

Availability.  This  criterion  often  weeds  out  a  lot  of  potential  indicators. 
Either  data  are  available  or  they  are  not.  If  the  data  from  the  intervention 
community  are  not  available,  those  responsible  for  the  evaluation  will  have  to 
decide  if  the  project  is  able  to  collect  the  data  or  collaborate  with  a  group  to  do 
so.  For  example,  it  is  likely  that  the  number  of  hours  that  physical  activity 
facilities  are  open  (Table  8.5)  is  not  currently  collected  in  a  community.  If, 
however,  a  main  goal  of  a  project  is  to  increase  physical  activity  by  increasing 
access  to  facilities,  the  project  may  consider  collecting  this  information  itself. 
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Cost.  Those  responsible  for  evaluation  must  examine  how  much  time, 
effort,  and  money  will  go  into  collecting  or  analyzing  indicator  data.  In  some 
cases,  data  are  already  being  collected  on  community-level  indicators.  For 
example,  it  is  likely  that  someone  collects  data  on  highway  funds  for  non-vehicle 
transportation  (Table  8.5).  The  challenge  is  to  find  who  does  collect  it. 
Precollected  data,  or  archival  data,  may  be  available  from  health  departments, 
parks  and  recreation  departments,  transportation  departments,  city  councils, 
county  commissions,  school  districts,  or  voluntary  organizations  (see  Table  8.2). 

Reliability  and  Validity.   Reliability  and  validity  are  scientific  standards  used  to 
judge  the  quality  of  data.  Some  aspects  of  accuracy  and  sensitivity  are  included 
in  the  definitions  of  reliability  and  validity.  A  reliable  assessment  produces  the 
same  estimates  over  and  over  again.  For  example,  no  matter  how  often  a  scale 
is  used  to  measure  weight,  it  usually  comes  up  with  approximately  the  same 
weight  for  the  same  object.  Some  assessments  of  measurement  reliability  are 
complicated.  For  example,  police  citations  for  selling  tobacco  products  to 
minors  may  be  a  good  marker  of  the  level  of  sales  to  minors.  However,  if  more 
police  are  hired,  an  increase  in  citations  may  not  be  due  to  increased  sales  but  to 
increased  ability  to  enforce  the  law.  the  measurement  system  changed. 

Measurement  validity  is  the  extent  to  which  an  instrument  measures  what  it  is 
intended  to  measure.   For  example,  a  ruler  is  a  valid  measure  of  length.  A  ruler 
is  not  a  valid  measure  of  weight.  This  example  is  obvious,  but  some  other 
assessments  of  measurement  validity  are  not  so  obvious.  For  example,  is 
supermarket  shelf-space  dedicated  to  low-fat  milk  a  valid  measure  of  low-fat  milk 
consumption?  Is  it  a  valid  measure  for  a  community's  diet.  Arguments  can  be 
made  for  and  against  the  validity  of  this  measure. 

Some  recommended  community-level  indicators  are  of  known  reliability  and 
validity.  For  example,  the  prevalence  of  tobacco  use  as  measured  by  some 
behavioral  surveys  is  both  valid  and  reliable.  Many  community-level  indicators 
we  provide  here  are  new  and  are  being  pilot-tested.  For  example,  the 
percentage  of  health  care  providers  that  routinely  advise  patients  to  exercise  is  a 
relatively  untested  indicator.  Such  indicators  of  the  cardiovascular  health  of  a 
community  are  of  undetermined  reliability  and  validity. 

Most  community-level  indicators  will  be  collected  in  different  ways  in  each 
community.  The  reliability  and  validity  of  these  measures  will  remain 
undetermined,  because  the  time  and  money  required  to  validate  them  would 
preclude  their  use.  Such  measures  may  still  provide  very  useful  indicators  of  the 
effects  a  program  is  having  on  the  community. 
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What  is  the  message?  Each  program  may  wish  to  collect  data  on  a  variety  of 
community-level  indicators.  Programs  could  collect  data  on  some  indicators  of 
known  reliability  and  validity  in  order  to  compare  data  from  their  community  to 
State  or  national  norms.  Programs  will  also  benefit  from  collecting  data  on 
some  indicators  of  unknown  reliability  if  these  indicators  may  be  particularly 
sensitive  to  program  efforts. 

The  Selection  Process 

As  recommended  earlier  in  the  chapter,  the  initiative  may  wish  to  form  a  task 
force  to  guide  the  process  of  selecting  indicators.  The  selection  process  includes 
(a)  selecting  an  initial  long  list  of  candidate  indicators,  (b)  evaluating  each 
candidate  indicator's  quality  and  feasibility,  and  (c)  creating  a  short  list  of 
candidate  indicators. 

Creating  a  Long  List  of  Candidate  Indicators.  When  choosing  what,  if  any, 
community-level  indicators  to  collect,  consider  (a)  what  members  and  leaders  of 
the  community  partnership  want  to  know,  (b)  requirements  from  grantmakers 
and  others  about  what  evaluation  questions  should  be  addressed,  and  (c) 
resources  available  to  address  these  questions. 

With  these  considerations  in  mind,  the  task  force  first  selects  a  long  list  of 
indicators  from  Tables  8.4,  8.5,  and  8.6;  also,  the  task  force  may  choose  to 
develop  or  add  other  indications  that  are  not  listed.  Members  of  the  initiative, 
leadership,  and  grantmakers  should  help  guide  the  selection  process. 


Factors  to  consider  when  choosing 
CVD  community-level  indicators 

Which  sectors  are  involved  in  the  community  effort? 

•  Schools. 

•  Worksites. 

•  Public  and  private  agencies. 

•  Religious  institutions. 

•  Community  at  large. 
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Factors  to  consider  when  choosing 
CVD  community-level  indicators  (Continued) 

Which  strategies  are  being  used  to  promote  changes  in 
the  community? 

•  Providing  information  and  skill  building. 

•  Barrier  modification  and  environmental  change. 

•  Policy  and  regulation. 

On  which  behavioral  risk  factors  does  the  program  focus? 

•  Tobacco  control  and  reduction  in  use. 

•  Increased  physical  activity. 

•  Improved  diet  and  nutrition. 


The  task  force  should  include  a  variety  of  indicators  on  its  long  list.  They  should 
include  indicators  from  sectors  of  the  community  that  are  creating  changes  to 
promote  cardiovascular  health.  For  example,  if  businesses  are  active  in 
program  efforts,  the  project  may  collect  worksite-based  indicators  such  as  the 
percentage  of  worksites  that  allow  staff  to  participate  in  physical  activity  during 
work  hours  (Table  8.5).   It  is  also  important  to  include  indicators  that  measure 
strategies  used  by  the  program.  For  example,  a  program  that  puts  most  of  its 
efforts  into  policy  change  should  select  indicators  that  measure  policy  changes, 
such  as  the  number  of  existing  tobacco  control  ordinances  (Table  8.4).  Finally,  a 
program  focusing  on  only  one  of  the  major  risk  factors  for  CVD  should  select 
indicators  within  that  domain.  For  example,  a  program  that  seeks  to  reduce 
CVD  by  increasing  physical  activity  may  limit  itself  to  indicators  related  to 
physical  activity  (Table  8.5). 

Evaluating  Candidate  Indicators.  After  a  long  list  of  candidate  indicators  is 
identified,  those  responsible  for  evaluation  review  each  candidate  indicator.  The 
task  force  may  wish  to  create  worksheets  on  which  members  can  note  the 
strengths  and  weaknesses  of  each  candidate  indicator.  The  task  force  may 
assign  several  indicators  to  each  member.  These  members  can  make  the  phone 
calls  necessary  for  collecting  information  about  quality  and  feasibility  of  the 
indicators  and  report  the  results  back  to  the  group.  In  addition  to  collecting 
information  on  quality  and  feasibility,  the  worksheets  can  provide  space  for  data 
issues  (who  will  secure  the  data,  how  it  might  be  collected,  or  who  could  use  the 
data)  and  tailoring  (depending  on  the  program  and  evaluation  goals,  the 
indicator  may  need  to  be  revised  or  require  further  definition).  The  task  force 
should  spend  some  time  brainstorming  on  who  would  be  good  contacts  for 
information  about  indicators.  Figure  8.1  gives  an  example  of  a  worksheet  on 
evaluating  a  candidate  indicator. 
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Figure  8.1 
Example  worksheet  for  a  candidate  indicator 


Candidate  Indicator  #6:    "Proportion  of  shelf  space  for  lower-fat  milk 
(2%,  1%,  and  skim)  to  whole  milk  in  grocery  stores" 

Task  force  member:   Jane  Doe 

Name,  affiliation,  phone  number  of  contacts: 
John  Smith,  Brand  X  Supermarket,  000-0000. 
Sally  Jones,  State  Dairy  Counc'I,  (000)  000-0000. 

I.  Quality 

The  data  may  be  sensitive  to  and  accurately  reflect  changes  in  barriers  to 
healthy  nutrition;  could  reflect  increased  local  demand  for  lower-fat 
products  because  of  program  efforts;  may  be  worth  trying. 

II.  Feasibility 

Can't  find  anyone  who  presently  collects  these  data.  Getting  data  for  this 
indicator  is  likely  to  require  in-store  observations.  This  would  be  time- 
consuming.  However,  an  initial  study  of  15  stores  in  8  communities  had 
the  same  power  as  a  telephone  survey  of  1,600  individuals,  at  a  fraction 
of  the  cost.  Could  explore  getting  high  school  home  economics  classes  to 
measure  this  for  class  credit. 

III.  Data  issues 

It  would  be  appropriate  for  either  community-based  programs  and  State 
health  departments  to  use  these  data,  although  collection  would  likely 
occur  at  the  community  level. 

IV.  Tailoring 

Decisions  would  have  to  be  made  concerning  how  many  stores  make  up  a 
representative  sample. 

Creating  a  Short  List  of  Candidate  Indicators.  With  information  for  each 
candidate  indicator  such  as  that  provided  above,  the  task  force  can  create  a 
short  list  of  indicators.  None  of  the  indicators  will  be  perfect — the  task  here  is  to 
find  the  best  indicators  possible.  The  quality  and  feasibility  of  each  candidate 
indicator  should  be  reviewed  and  compared  with  other  indicators.  The  quality 
criterion  consists  of  accuracy  and  sensitivity.  Sometimes  there  are  so  many 
flaws  that  the  data  are  inaccurate.  Seriously  flawed  indicators  should  be 
excluded  from  the  list  of  candidate  indicators.  Indicators  that  are  not  sensitive — 
those  that  do  not  correspond  to  program  efforts,  or  that  will  not  register 
program  effects — should  be  excluded  from  the  initial  list  of  candidate  indicators. 

The  criterion  of  feasibility  consists  of  availability  and  cost.  Many  indicators  will 
not  be  in  existing  databases.  If  they  are  potentially  important  measures,  the 
project  may  choose  to  collect  them,  or  encourage  another  group,  such  as  a  local 
university  or  volunteer  organization,  to  collect  the  data.  If  an  indicator  does  not 
correspond  to  project  efforts  and  is  not  readily  available,  it  should  be  excluded 
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from  the  list.  Some  indicators  may  require  on-site  observation,  such  as  usage  of 
bike  or  fitness  trails,  or  locally  created  surveys.   Indicators  that  require  more 
resources  than  are  feasible  should  be  excluded  from  the  initial  list  of  candidate 
indicators. 

Those  responsible  for  evaluation  will  end  up  with  a  short  list  of  candidate 
indicators.  The  evaluators  may  collect  data  on  each  one. 


Collecting,  Graphing,  and  Communicating  Data 

Tips  on  Collecting  Data 

Those  responsible  for  evaluation  will  have  to  determine  who  will  collect 
indicator  data,  how  often  it  will  be  collected,  and  how  the  data  will  be  stored.  If 
the  program  has  HIGH  resources  and  requirements  for  evaluation,  hired  outside 
evaluators  will  be  the  natural  choice.  If  the  program  has  LOW  resources  but  still 
wishes  to  collect  indicator  data,  one  or  several  members  of  the  program  may 
volunteer  to  collect  the  data.  Members  may  also  develop  innovative 
collaborations  with  groups  (such  as  4-H  clubs,  extension  agents,  community 
service  groups,  or  health  organizations)  who  may  do  the  on-site  data  collection. 

Table  8.2  provides  examples  of  sources  for  archival  data  on  community-level 
indicators  of  cardiovascular  health. 


Table  8.2 
Obtaining  CVD  community-level  indicators:    Some  resources 


CVD  Risk  Factor 


Potential  Resources 


Tobacco  Control 


Physical  Activity 


Diet 


Organizations:   Local  and  State  health  departments,  local  and 
State  education  departments,  local  school  districts,  local  and  State 
departments  of  revenue  and  tax  commissions,  American  Cancer 
Society,  American  Heart  Association,  CDC 

Organizations:   Local  and  State  health  departments,  local  and 
State  parks  and  recreation  departments,  local  and  State  education 
departments,  local  and  State  transportation  departments,  local 
school  districts,  American  College  of  Sports  Medicine,  CDC 

Organizations:  Local  and  State  health  departments,  local  and 
State  education  departments,  local  school  districts,  American 
Cancer  Society,  American  Heart  Association,  CDC 
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Those  responsible  for  evaluation  should  determine  how  often  the  data  will  be 
collected.  The  data  may  be  collected  monthly,  quarterly,  annually,  or  at  less 
regular  intervals.  Once  collected,  the  data  may  be  stored  in  file  folders  or  in 
computer  databases. 

Table  8.3  provides  a  sample  data  collection  form.  It  illustrates  how  data  might 
be  stored  for  the  number  of  smoking  cessation  programs  available  in  the 
community  by  sector  (3  years  of  baseline  established  by  interviews  with  health 
care  providers). 

Table  8.3 
Data  Form 


SECTOR 

Year 

1992 

1993 

1994 

1995 

1996 

1997 

Worksites 

1 

2 

3 

Schools 

0 

0 

1 

Health  Organizations 

2 

2 

2 

Others 

1 

2 

3 

Total 

4 

6 

9 

Whenever  possible,  the  initiative  should  collect  historical  baseline  data  for  3-5 
years  before  program  onset.  Baseline  data  are  measures  of  what  was 
happening  to  CVD  risks  in  the  community  before  the  community  partnership 
existed.  Three  to  five  years  of  baseline  data  will  tell  you  whether  the  indicator  for 
CVD  was  increasing  or  decreasing  when  the  program  began  its  efforts. 
Knowledge  of  these  trends  will  help  the  initiative  determine  what  impact  its 
efforts  might  have  on  CVD  in  the  community. 

Tips  on  Graphing  Data 

You  will  have  to  determine  who  will  graph  indicator  data  and  how  often  graphs 
will  be  updated.  If  the  program  has  HIGH  resources  and  requirements  for 
evaluation,  hired  evaluators  will  be  the  natural  choice.  If  the  program  has  LOW 
resources  but  still  wishes  to  collect  and  graph  indicator  data,  one  or  several 
members  of  the  program  may  volunteer  to  graph  the  data.  Members  may  also 
develop  innovative  collaborations  with  groups  (such  as  business  clubs,  university 
research  groups,  individuals  with  an  interest  in  CVD  reduction,  or  health 
organizations)  who  may  graph  data  for  the  initiative. 
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Tips  on  Presenting  Data 

Data  often  speak  louder  than  words.  Newspapers  often  print  local  data  about 
issues  of  national  importance.  Local  civic  organizations  often  seek  speakers  who 
can  present  data  on  the  well-being  of  the  community.  For  example,  if  a  project 
member  had  a  graph  of  the  prevalence  of  smoking  cessation  programs  in  the 
community  and  an  idea  of  how  many  people  smoke  in  the  community  (usually 
over  25%),  the  project  member  would  be  able  to  make  a  strong  case  that  the 
initiative  was  making  changes  to  increase  the  health  of  the  community.  For 
other  tips  on  communicating  data,  see  Chapter  11. 

Community-level  Indicators  —  A  Menu  of  Options 

Tables  8.4,  8.5  and  8.6  list  examples  of  potential  community-level  indicators  of 
cardiovascular  health.  Each  table  represents  one  of  the  major  risk  factors 
(tobacco  use,  physical  inactivity,  and  diet)  with  indicators  classified  by  three 
broad  intervention  strategies  (information  and  skill  building,  barrier  modification 
and  environmental  change,  and  policy  and  regulation).  These  lists  are 
comprehensive  but  not  exhaustive — initiatives  may  identify  others  that  best 
measure  their  efforts.  Each  indicator  will  have  to  be  tailored  to  meet  the  needs 
of  a  State  or  local  initiative.  For  example,  "Clean  indoor  air  policies"  (Table  8.4) 
does  not  specify  exactly  what  data  to  collect.  Evaluators  may  decide  that  "the 
percentage  of  all  public  buildings  with  clean  indoor  air  policies"  is  the  best 
indicator  of  the  efforts  of  the  CVD  prevention  initiative. 

Table  8.4 

Potential  CVD  community-level  indicators 

for  tobacco  control 

Information  and  Skill  Building 

•  Surveillance  data  on  tobacco  sales  to  minors 

•  Existence  of  smoking  control  programs  in  public  health  agencies 

•  Existence  of  smoking  cessation  campaigns  within  worksites 

•  Number  of  community  coalitions  for  tobacco  control 

•  Level  of  involvement  of  community  coalitions  in  tobacco  control 

•  Training  of  local  leaders  on  tobacco  advocacy 

•  Presence  or  absence  of  advertising  of  tobacco  paraphernalia  (clothing,  pictures,  etc.) 

•  Presence/absence  of  tobacco  control  advertising,  media  messages,  or  hotline 

•  Existence  of  smoking  cessation  services  within  (across)  settings  in  the  community 

•  Prevalence  of  tobacco  use  (See  Chapter  7) 
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Table  8.4 

Potential  CVD  community-level  indicators 

for  tobacco  control  (Continued) 


Barrier  Modification  and  Environmental  Change 

Percentage  of  restaurant  seats  within  no-smoking  sections 

Price  of  tobacco  products 

Percentage  of  worksites  with  no-smoking  areas 

Presence/absence  of  tobacco  vending  machines  in  restaurants  or  hotels 

Disappearance  of  tobacco  products  (store  inventory) 

Observations  in  smoking  and  non-smoking  areas 

Location  of  tobacco  vending  machines  in  restaurants,  hotels,  etc.  (i.e.,  are  they  by 
the  front  door  where  access  is  easier?) 

Per  capita  consumption  of  cigarettes 

Presence/absence  of  tobacco  advertising,  promotion,  or  free  distribution  of 
tobacco  products  in  the  community,  near  schools,  or  worksites 

Policy  and  Regulation 

Clean  air  laws  for  public  buildings,  restaurants,  private  worksites,  daycare, 
schools,  etc. 

Prohibition  on  use  of  tobacco  products  on  public  property  (e.g. ,  venues,  sports 
stadiums,  housing,  and  worksites) 

Fines  or  loss  of  license  for  violations 

Excise  tax  on  tobacco  products  (local  option) 

Existence  of  community  tobacco  control  ordinances 

Excise/license  fee  for  tobacco  sales 

Vending  machine  regulations  in  communities 

Restrictions  on  advertising  for  tobacco  products  on  public  property,  or  near  youth 
areas  (e.g.,  schools) 

Enforcement  of  Synar  ordinances  (no  sales  to  minors)  throughout  the  community 

Restriction  on  tobacco  product  ads  in  communities 

School  policies  banning  smoking 

Evidence  of  enforcement  (e.g.,  fines  and  litigation)  of  tobacco  ordinances  and  laws 

Youth  access  laws  in  communities:  crime  for  sale  to  youth  and  enforcement  of  laws 
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Table  8.5 
Potential  CVD  community-level  indicators  for  physical  activity 

Information  and  Skill  Building 

•  Percentage  of  time  in  physical  educational  (PE)  classes  spent  on  "lifetime"  exercises 

•  Percentage  of  health  care  providers  that  routinely  advise  patients  to  exercise  more 

•  Availability  of  materials  in  worksites  linking  sedentariness  with  CVD 

•  Prevalence  of  physical  activity  (see  Chapter  7) 

•  Number  of  media  reports  (TV,  radio,  print)  linking  physical  inactivity  to  CVD;  also, 
amount  of  space  or  length  of  time  of  report 

Barrier  Modification  and  Environmental  Change 

•  Observation  of  usage  (in  malls  or  on  trails) 

•  Percentage  of  walking  trails  (or  miles  of  walking  trails  per  capita) 

•  Percentage  of  bike  lanes  (or  ratio  of  bike  lanes  to  streets  or  miles  of  bike  lanes  per 
capita) 

•  Number  of  physical  activity  facilities  (fields,  gyms,  pools,  etc.)  per  capita  in  schools 

•  Membership  in  physical  activity  organizations  (Y's,  health  clubs,  etc.) 

•  Availability  of  facilities  (fields,  gyms,  pools,  etc.)  to  community  members 

•  Number  of  physical  activity  facilities  (fields ,  gyms ,  pools ,  etc . )  per  capita  in  worksites 

•  Hours  that  physical  activity  facilities  (fields,  gyms,  pools,  etc.)  are  open 

•  Number  of  worksites  that  sponsor  teams,  jogging  groups,  mall  walking,  sporting 
events,  etc. 

•  Number  of  worksite/offices,  etc. ,  that  allow  staff  to  participate  in  physical  activity 
during  working  hours 

•  Tracking  of  targeted  items  (such  as  sales  or  rental  of  sports  equipment  and  videos) 

•  Capacity  of  physical  activity  facilities  (fields,  gyms,  pools,  etc.) 

•  Acres  of  parks  or  recreational  space  (per  total  community  acres  or  per  capita) 

•  Census  of  use  of  physical  activity  facilities  (worksite,  community,  etc.) 

•  Number  of  physical  activity  events  in  the  community  per  year 

•  Presence  or  absence  of  formal  worksite  policies  that  support  physical  activity  (e.g. , 
flextime  or  longer  breaks/lunches) 

Policy  and  Regulation 

•  Highway  funds  for  non-vehicle  transports  (ISTEA):  amount  of  money  and  number 
of  projects 

•  Local/State  policy  to  include  physical  activity  in  curriculum  of  public  schools  K-12 

•  Include  green-walk/bike  ways  in  zoning/rezoning  requirements 

•  Amount  or  percentage  of  local  budget  per  capita  devoted  to  physical  activity/ 
recreation 

•  Insurance  reimbursement  for  physical  activity  services  and  counseling  in  health 
organizations 

•  Presence  or  absence  of  policies  promoting  inclusion  of  recreation  facilities  with  new 
construction/remodeling/restoration 

•  Local/State  policy  to  include  physical  activity  in  curriculum  of  private  schools 

•  Evidence  that  local  zoning  board  considers  health  (CVD)  in  its  decisions 

•  Extent  of  public  schools  (K-12)  that  include  lifetime  activity  skills 
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Table  8.6 
Potential  CVD  community-level  indicators  for  diet 

Information  and  Skill  Building 

•     "  Point-of-purchase "  information  provided  (e .  g . ,  in  cafeterias ,  vending  machines , 
stores,  and  restaurants) 

Healthy  menus  in  schools,  worksites,  and  other  places 
Training  for  cooks  in  cafeterias  (schools,  worksites,  and  restaurants) 
Presence  or  absence  of  shopping  tours/demonstrations/food  sampling  in 
supermarkets 

Presence  or  absence  of  low-fat  cooking  via  the  media  (TV)  offered  on  Saturday  or 
Sunday 

Number  of  contests  for  diet  change  within  the  community 
Presence  or  absence  of  food  pyramid  charts  in  "learning"  environments 
Number  of  media  reports  (TV,  radio,  and  print)  linking  nutrition  to  CVD;  also, 
amount  of  space  or  length  of  time  of  report 

Existence  of  food  preparation  classes  within  (across)  settings  in  the  community 
Prevalence  of  overweight  individuals  (see  Chapter  7) 

Barrier  Modification  and  Environmental  Change 

Presence/absence  of  healthy  food  alternatives  in  vending  machines  in  schools, 

worksites,  and  public  buildings 

Bar  codes  sales  data  on  foods  that  prevent  or  increase  risks  for  CVD 

Proportion  of  low-fat  items  in  schools,  via  menu  analysis  of  use  of  indicator  items 

such  as  low-fat  milk  (also  use  for  worksite,  restaurants,  etc.) 

Inventory  control  data  (e.g.,  schools  and  worksite  cafeterias)  for  food  usage 

Proportion  of  low-fat  items  in  stores  (via  use  of  indicator  items  such  as  low-fat  milk) 

Proportion  of  shelf  space  in  grocer  stores  of  low-fat  foods 

Observations  of  patrons  at  grocery  stores  (e.g.,  percentage  choosing  low-fat 

foods) 

Number  of  "fast  food"  restaurants  per  capita 

Presence/absence  of  farmer's  markets,  coops,  green  grocers 

Policy  and  Regulation 

Number  or  percentage  of  schools  with  lunch  menus  consistent  with  nutritional 

guidelines 

Number  or  percentage  of  schools  with  policies  requiring  alternatives  to  snack/ 

junk  food  in  vending  machines 

Proportion  of  publicly  funded  food  programs  that  follow  nutritional  guidelines 

Index  of  local  district  policies  regarding  nutrition  at  school,  day  care,  after-school, 

and  weekend  events 

Commodity  food  programs:  presence  or  absence  of  low-fat  foods 

Presence  or  absence  of  State  guidelines,  monitoring  guidelines,  and  enforcement 

of  nutrition  in  schools  (FDA  or  other) 

Presence  or  absence  of  food  policies  for  special  populations  (older  adults. 

Women,  Infants,  and  Children  [WIC]) 

Third-party  reimbursement  for  dietary  counseling 

Monitoring  or  regulating  claims  made  by  food  retailers  regarding  the  "heart  healthiness" 

of  food  items  they  sell 

Adoption  of  formal  policy  that  supports  healthy  eating  at  worksites 
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9 


Conducting  Interviews  with 
Key  Participants 


Conducting  semistructured  interviews  with  key  participants  helps  understand 
the  process,  intermediate  outcomes,  and  distal  outcomes  of  the  initiative. 
This  qualitative  component  is  designed  to  identify  critical  events  in  the  history  of 
the  initiative  and  their  meaning  for  participants.  Interviews  help  answer  the 
following  key  evaluation  question:   "What  critical  events  seem  to  spur  rates  of 
community  change?" 


Interviews  with  key  participants:    What  and  why? 
What  are  they? 

•  A  way  of  learning  about  important  events  in  the  life  of  the  initiative. 
Why  do  it? 

•  To  find  out  what  events  were  important  and  why  they  were  important. 

•  To  help  identify  factors  that  affected  the  initiative's  success. 

•  To  help  identify  negative  side  effects. 

•  To  provide  a  history  of  the  initiative. 

•  To  obtain  insights  that  can  be  used  in  planning  and  renewal  efforts. 


Interviews  with  key  participants  provide  qualitative  information  about  important 
events  in  the  initiative  and  their  meaning.  Key  participants  are  people  who  have 
been  with  the  coalition  since  its  inception  or  who  have  been  most  actively 
involved.  They  are  frequently  project  staff  and  active  members  of  the  coalition 
such  as  committee  chairpersons.  These  key  participants  are  interviewed  to  gain 
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their  insights  into  important  events  in  the  life  of  the  initiative,  lessons  learned, 
and  future  directions. 

Hindsight  is  20-20  vision;  sometimes  one  must  reflect  back  over  the 
development  of  a  community  initiative  to  identify  events  that  proved  to  be 
important.  Semistructured  interviews  gather  information  about  critical  events  in 
the  unfolding  of  the  initiative  from  the  perspective  of  those  most  involved. 
Important  events  that  are  revealed  in  these  interviews  can  help  the  initiative 
during  transitions  such  as  changes  in  staff,  the  board  of  directors,  and  the 
initiative's  mission. 

This  chapter  describes  how  to  conduct  these  interviews  and  how  interviews  can 
help  identify  and  explore  critical  events. 


Interviews  with  key  participants:    How  to's 

•  Staff  and  leadership  generate  names  of  active  participants. 

•  Interviewers  are  selected  to  conduct  interviews. 

•  A  designated  interviewer  combines  information  from  the  interviews 
in  a  report. 

•  The  initiative  and  funders  receive  a  copy  of  the  report. 


Those  assuming  responsibility  for  this  component  of  the  evaluation  ask  project 
staff  and  others  for  the  names  of  several  persons  who  have  made  large 
contributions  in  time  and  effort  to  the  initiative  and  who  have  extensive 
knowledge  about  the  initiative  and  the  community.  Evaluators  conduct 
individual  phone  or  face-to-face  interviews  with  four  to  eight  of  the  most 
frequently  nominated  persons.  The  interviews  help  identify  critical  events  and 
explore  why  they  were  important.  Key  participants  identify  the  critical  events. 
Interviewers  ask  questions  about  each  event.  Some  key  questions  are  listed 
below. 


90  CVD  Evaluation  Handbook 


Interviews  about  critical  events:     Some  key  questions 

What  are  the  critical  events  in  the  history  of  the  initiative? 

•  Why  was  the  event  important? 

•  What  was  the  context  for  the  event? 

•  What  key  actions,  actors,  and  other  resources  were  required? 

•  What  barriers  and  resistance  were  encountered? 

•  What  were  the  consequences  of  the  event  for  the  initiative  and  the 
community? 

What  lessons  did  the  initiative  or  individual  learn? 

What  future  directions  should  be  taken  by  the  initiative? 


Project  evaluators  conduct  the  interviews  and  combine  participant  comments  in 
a  report.  Each  interview  lasts  about  90  minutes.  The  questions  attempt  to 
identify  critical  events,  consequences  for  the  initiative  and  the  broader 
community,  and  future  directions  for  the  initiative.  The  interview  could  be 
audiotaped,  with  consent  of  the  interviewee,  to  aid  in  preparing  the  report.  In 
the  report,  evaluators  combine  comments  from  all  participants  who  reported  on 
a  particular  event.  Comments  are  not  linked  to  individual  interview  participants, 
so  a  degree  of  anonymity  is  maintained.  The  initiative  can  use  the  report  to 
provide  a  history  of  the  initiative,  aid  in  funding  efforts,  assist  in  planning  for  the 
future,  and  better  understand  the  initiative. 

Key  participant  interviews  could  be  conducted  every  several  years  or  in  the  last 
year  of  a  grant  period.  More  regular  interviews  would  yield  valuable  information, 
such  as  the  current  barriers  faced  by  the  initiative,  that  could  help  the  initiative  in 
troubleshooting  how  to  complete  current  projects. 
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Adjusting  the  System  to  Meet  Your  Needs 


Consider  the  following  when  deciding  how  to  gather 
qualitative  information  about  the  partnership 

What  members  and  leaders  of  the  community  partnership  want  to  know. 

Requirements  from  grantmakers  and  others  about  what  questions  should 
be  addressed. 

Resources  available  to  address  these  questions. 


For  community  partnerships  with  HIGH  resources  and  requirements  for 
evaluation,  consider  doing  annual  (or  even  more  frequent)  interviews  with  six  to 
eight  key  participants  in  the  partnership.  This  might  presume  the  availability  of 
active  coalition  members  and  staff  to  talk  about  the  partnership  for  about  2 
hours  each,  20  hours  or  more  time  of  a  person  with  basic  skills  in  interviewing, 
and  secretarial  time  in  preparing  a  report. 

For  community  partnerships  with  LOW  resources  and  requirements,  consider 
fewer  interviews  or  a  less  formal  approach  to  interviewing.  Recommended 
strategies  for  conducting  interviews  with  key  participants  in  partnerships  with 
limited  resources  are  listed  on  the  following  page. 


Recommended  Strategies  for  Interviewing  on  a  Shoestring  Budget 

•  Conduct  interviews  with  only  three  to  four  key  participants  in  the  coalition. 

•  Summarize  the  information  informally  in  a  few  statements  about  broad 
categories  such  as  key  events  and  lessons  learned. 

•  Instead  of  using  individual  interviews,  collect  information  in  informal 
listening  sessions  with  a  group  of  representative  members.  Flip  charts  or 
bulletin  boards  can  be  used  to  capture  member  input  on: 

Key  events  (and  dates): 

Why  they  are  important 

Key  actors 

Barriers  and  resistance 

Consequences  for  the  coalition  and  the  community. 
Lessons  learned. 
Future  directions  for  the  partnership. 
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Materials  on  Conducting  Interviews 

This  section  provides  a  checklist  and  materials  for  conducting  interviews  with 
key  participants.  The  format  for  the  interview  includes  a  list  of  questions.  This 
section  also  includes  a  sample  of  a  critical  events  report  for  a  community 
initiative  to  reduce  the  risks  for  CVD. 

Interview  Checklist  and  Format 

Table  9.1  provides  a  checklist  for  preparing  for  the  interviews,  conducting  the 
interviews,  and  preparing  the  leport.  Figures  9.1  and  9.2  provide  samples  of 
formats  for  the  critical  interview,  including  a  cover  sheet  with  introductory 
questions  and  specific  questions  for  each  critical  event. 

Table  9.1 
Checklist  for  conducting  interviews 


Before  the  Interview 

•  Make  a  copy  of  the  cover  sheet  on  which  to  list  the  critical  events  (see 
Figure  9.1). 

•  Review  the  list  of  questions  for  each  event  (see  Figure  9.2). 

•  Prepare  pages,  blank  or  with  headings,  to  record  responses  from  key 
participants  or  use  a  tape  recorder  to  record  conversations. 

•  Look  up  the  dates  for  some  major  events  (e.g. ,  grant  awards  or  hiring  staff) 
to  use  as  a  benchmark  when  a  participant  has  difficulty  remembering  the 
date  of  a  critical  event. 

•  For  each  participant,  call  in  advance  to  arrange  a  convenient  time  for  a  90- 
minute  interview. 

During  the  Interview 

•  State  the  purpose  of  the  interview  and  remind  the  key  participant  that  the 
interview  is  intended  to  focus  on  the  events  critical  to  the  entire  life  of  the 
initiative. 

•  Describe  the  format  for  the  interview  and  approximately  how  long  it  should 
take  (approximately  1.5  hours). 

•  Ask  the  key  participant  to  identify  the  critical  events,  such  as  when  key  staff 
were  hired,  noting  an  approximate  month  and  year  when  the  event  took 
place. 

•  Discuss  each  event  using  the  interview  questions  (see  Figure  9.2). 

•  After  each  event  is  discussed,  discuss  overall  lessons  and  future  directions 
(see  Figure  9.2). 
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Table  9.1 
Checklist  for  conducting  interviews  (Continued) 

During  the  Interview  (cont.) 

•  Close  the  interview  by  thanking  the  participant  for  his  or  her  time  and  offer 
to  send  a  copy  of  the  completed  report. 

Preparing  the  Report 

•  Review  notes  from  the  interviews  (or  transcribe  interviews  if  a  tape  recorder 
was  used). 

•  Combine  lists  of  critical  events  from  all  interviews. 

•  Combine  responses  to  questions  for  common  critical  events. 

•  Summarize  responses  for  each  critical  event.  Use  the  language  of  the  key 
participant. 

•  Distribute  report  to  project  staff  and  leadership  and  other  interested 
constituents. 
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Figure  9.1 
Cover  sheet  for  the  Critical  Events  Interview 

Initiative:  Key  Participant: 


Participant's  Position  (e.g.,  health  department  official,  parent,  teacher): 


Participant's  Involvement  with  the  Initiative  (e.g.,  founding  partner, 
member,  chair): 


Date  of  Interview:  Interviewer: 

Interview  Process:    Ask  introductory  questions:    What  key  events  or  incidents 
were  critical  to  the  initiative's  development?  Its  major  accomplishments  or 
successes?  Its  setbacks  or  challenges? 

After  listening  to  the  key  participant  and  taking  notes  on  this  page,  identify  the 
several  particularly  important  events  (usually  6-12  discrete  events).  State  these 
to  the  participant,  asking  for  agreement  about  them.  Each  identified  critical 
event  will  then  be  considered  separately. 


Notes: 


Date  (month,  year)  Event 
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Figure  9.2 
Interview  questions 


For  each  Critical  Event,  ask  about: 

DATE  of  the  event:  (month,  year) 

RATIONALE:   (Why  was  this  event  particularly  important?) 

CONTEXT  OR  CONDITIONS:   (What  was  going  on  at  the  time  of  the 
event?  What  made  the  conditions  right  for  this  to  happen?) 

KEY  ACTIONS  AND  ACTORS:   (What  key  actions  brought  about  the 
critical  event?  Who  were  the  key  actors?) 

KEY  RESOURCES :  (What  key  resources  [e .  g .,  people ,  financial  resources , 
political  influence]  were  used  to  bring  about  the  critical  event?  How  were 
these  resources  used  to  overcome  barriers  and  resistance?) 

BARRIERS  AND  RESISTANCE:   (Were  the  group's  actions  met  with 
barriers  or  resistance?  What  types  of  barriers?  Who  resisted?) 

CONSEQUENCES  for  the  initiative:   (What  were  the  consequences  or 
results  of  the  critical  event  for  the  initiative?) 

CONSEQUENCES  for  the  community:   (What  were  the  consequences  for 
the  community?) 

After  all  identified  events  have  been  discussed,  ask  about: 

OVERALL  LESSONS:  (Overall,  what  lessons  have  you  learned  from  your 
involvement  with  the  initiative?  What  lessons  have  you  learned  from  the 
initiative's  attempts  to  define  and  act  on  its  mission?) 

FUTURE  DIRECTIONS:  (What  issues  does  the  initiative  face  in  the  future? 
What  challenges  should  be  addressed?) 


Sample  Report 

This  chapter  concludes  with  a  sample  report  on  critical  events.  A  Critical  Events 
Report  includes  a  list  of  key  participants  interviewed,  a  list  of  critical  events  that 
were  identified,  and  a  narrative  for  each  critical  event.  The  narrative  describes 
the  responses  of  the  interviewees  when  asked  questions  about  each  critical 
event.  When  two  or  more  key  participants  identified  the  same  event,  the 
responses  are  combined  in  the  report.  Figure  9.3  provides  an  example 
narrative  for  one  critical  event  for  Kansas  LEAN,  a  state  wide  coalition  to 
prevent  CVD. 
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Figure  9.3 
Sample  Critical  Events  Report  for  an  initiative  to  prevent  CVD 

List  of  Key  Participants  Interviewed 

•  Director,  Kansas  LEAN. 

•  Senior  Program  Officer,  Kansas  Health  Foundation. 

•  Director,  Bureau  of  Office  of  Health  Promotion  and  Chronic  Disease, 
Kansas  Department  of  Health  and  Environment. 

•  Official,  County  Extension  Office. 

•  Official,  Kansas  Wheat  Commission. 

List  of  Critical  Events  in  the  Life  of  the  Kansas  LEAN  Initiative 

1.  Initial  formation  of  Kansas  LEAN  (fall  1988). 

2.  Development  of  Partnerships  and  Task  Forces  (spring  1989). 

3.  Collaboration  with  Childcare  Association  (late  spring  1989). 

4.  Kansas  Health  Foundation  awards  the  Kansas  LEAN  grant  (July  1990). 

5.  Hiring  Kansas  LEAN's  Director  (July  1990). 

6.  Developing  the  Child  Care  Nutrition  Curriculum  (September  1990-April 
1994). 

7.  Action  planning  (October  1990-April  1991). 

8.  The  first  Annual  Kansas  LEAN  Partners'  Meeting  (November  1991). 

9.  Kansas  LEAN  facilitates  the  USDA  Food  Safety  education  project  (winter 
1991). 

10.   Kansas  LEAN  awarded  USDA  technical  assistance  grant  to  conduct  the 
CDC  Dietary  Intake  Survey  (completed  summer  1993). 

Critical  event:     7.  Action  Planning 

DATE  of  the  Event:   October  1990-April  1991 

RATIONALE:  The  action  plan  kept  us  on  a  time  line.  Things  were  no  longer  pie 
in  the  sky.  We  put  up  time  lines  that  gave  us  something  to  go  against.  The 
action  plans  are  very  much  a  working  document. 

CONTEXT  OR  CONDITIONS:  The  initiative  had  spent  a  year  talking  in 
generalities.  We  were  ready  to  get  down  to  action.  We  were  holding  press 
conferences  that  made  a  commitment  to  the  public  to  do  something.  We  took 
all  the  change  objectives  and  laid  out  our  action  plan  by  Task  Forces.  Each  Task 
Force  met  to  further  develop  the  action  plan.  Most  of  us  were  not  used  to  being 
in  a  true  collaboration.  The  diversity  of  the  group  required  that  we  figure  out 
how  people  fit.  If  we  didn't  define  roles,  we  would  lose  people. 
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KEY  ACTIONS  AND  ACTORS:  The  director  and  staff  were  key  leaders.  They 
forced  us  to  decide  what  we  were  going  to  do.  The  monthly  brainstorming  was 
very  helpful.  The  survey  of  importance  and  feasibility  was  critical.  The  survey 
gave  all  the  Task  Force  members  and  partners  a  chance  to  get  involved. 

KEY  RESOURCES:  The  task  forces  still  had  control  over  what  would  be 
pursued.  The  University  of  Kansas  evaluation  group  and  Kansas  LEAN  staff 
were  very  helpful  in  providing  technical  support. 

BARRIERS  AND  RESISTANCE:  Because  people  had  experienced  other  types 
of  strategic  planning,  some  eyes  rolled  when  we  started.  Some  thought  it  would 
not  produce  anything  useful.  Eventually  people  believed  that  this  planning 
process  would  be  different. 

CONSEQUENCES  (for  the  initiative):  The  strategic  plan  made  the  initiative 
members  feel  that  we  were  finally  underway.  It  helped  with  public  perception. 
We  now  had  substance.  This  is  what  we  are  and  what  we  do.  It  brought  the 
initiative  together  on  its  priorities.  For  example,  the  emphasis  on  early  nutrition 
made  it  easier  for  people  to  see  that  children  were  a  priority  for  the  initiative.  It 
also  made  it  much  easier  to  talk  with  potential  funders. 

CONSEQUENCES  (for  the  community):  Strategic  planning  got  programs  in 
place.  Without  it,  we  might  not  have  began  to  work  on  the  child  care  curriculum. 
The  action  plan  put  programs  out  there  that  weren't  there  before.  Action 
planning  also  helped  us  to  pull  in  new  partners.  As  the  plan  grew,  additional 
partners  become  obvious.  Partners  could  see  how  they  could  be  involved  with 
the  initiative  and  its  mission. 
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Using  the  Evaluation  System  to 
Answer  Key  Questions 


This  chapter  shows  how  information  obtained  from  this  comprehensive 
evaluation  system  can  be  interpreted  to  address  significant  evaluation  questions. 
In  order  for  an  evaluation  system  to  be  useful,  it  must  answer  key  questions  of 
relevant  audiences.   Listed  below  are  several  questions  that  appear  to  be  of 
interest  to  community  leadership,  grantmakers,  and  the  community  at  large. 


Evaluation  questions  of  possible  interest 

Was  the  community  mobilized  to  reduce  risks  for  CVD?  (Chapter  4) 

What  changes  in  the  community  resulted  from  the  initiative? 
(Chapter  4) 

Is  there  a  change  in  behavior  related  to  risks  for  CVD?  (Chapter  7) 

Were  members  satisfied  with  the  partnership?  (Chapter  5) 

Were  the  community  changes  important  to  reducing  risks  for  CVD? 
(Chapter  6) 

Does  the  initiative  have  a  community-level  outcome  related  to  the  risks  for 
CVD?  (Chapter  8) 

Is  community-level  outcome  related  to  changes  facilitated  by  the  initiative? 
(Chapters  4  and  8) 

What  critical  events  seemed  to  spur  rates  of  community  change? 
(Chapters  4  and  9) 


This  chapter  describes  how  the  measurement  system  can  be  used  to  answer 
many  of  these  questions  of  interest. 


CVD  Evaluation  Handbook 


101 


Was  the  Community  Mobilized  to  Reduce  the  Risks 
for  CVD? 

The  question  of  community  mobilization  is  important  for  collaborative 
partnerships  that  serve  as  catalysts  for  community  change.  Measures  of 
community  action  capture  community  mobilization.  Community  actions  are 
actions  taken  by  staff,  leadership,  volunteers,  and  community  members 
associated  with  the  initiative  to  facilitate  new  or  modified  programs,  policies, 
and  practices  relevant  to  the  mission  (see  Chapter  4  and  Appendix  A). 

Figure  10.1  illustrates  data  on  community  mobilization.  It  shows  community 
actions  by  members  and  staff  of  a  hypothetical  initiative  designed  to  reduce  risks 
for  CVD.  Each  community  action  is  a  different  event,  such  as  meeting  with 
school  food  service  personnel  about  reducing  the  fat  content  of  school  lunches. 
Community  actions  are  graphed  cumulatively  (i.e.,  new  events  are  added  to  the 
previous  events).  With  this  type  of  data,  a  flat  line  suggests  no  activity;  the 
steeper  the  slope  of  the  line,  the  more  activity.  This  type  of  graph  is  used 
because  it  effectively  shows  the  developmental  process  of  community 
mobilization. 

Figure  10.1 

A  hypothetical  coalition  to  prevent  CVD: 

Community  actions  taken* 
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'Examples  of  community  actions  include  events  such  as  meeting  with  school  food  service  per- 
sonnel about  reducing  the  fat  content  of  school  lunches,  holding  a  community  forum  to  hear 
citizen's  concerns  about  secondhand  smoke,  and  meeting  with  staff  of  the  department  of  trans- 
portation to  discuss  building  bike  paths. 
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For  this  hypothetical  coalition  (Figure  10.1),  during  the  first  year,  the  graph 
depicts  a  flat  line,  showing  no  activity.  An  increase  in  activity  is  shown  after  the 
grant  is  awarded  and  staff  is  hired.  Sharp  increases  in  actions  are  also  noted 
after  the  completion  of  planning  and  the  hiring  of  an  additional  staff  person. 
Some  leveling  off  of  activity  is  shown  in  the  spring  1991  with  another  increase 
in  the  last  few  months  of  data  collection.  Overall,  the  data  show  a  steady  level  of 
community  action  that  was  maintained  over  a  two-year  period;  thus,  the  data 
suggest  that  the  community  and  staff  were  mobilized  to  bring  about  community 
change. 


What  Changes  in  the  Community  Resulted  from  the 
Initiative? 

Community  change  is  defined  as  new  or  modified  programs,  policies,  or 
practices  related  to  the  mission  (see  Chapter  4  and  Appendix  A).  Figure  10.2 
shows  the  pattern  of  community  change  for  the  Statewide  initiative  Kansas 
LEAN  to  reduce  the  risks  for  CVD  and  some  cancers.  Kansas  LEAN  has  the 
primary  mission  of  reducing  consumption  of  dietary  fat.  Illustrative  community 
changes  for  Kansas  LEAN  included  developing  nutrition  information  for 
teachers  and  day  care  providers  (new  program)  and  collaborating  with  a  local 
supermarket  chain  to  implement  shelf  prompts  and  price  reductions  to  increase 
purchases  of  lower  fat  food  items  (new  program). 

Figure  10.2  shows  steady  growth  in  community  change  over  a  three-year 
period.  Community  change  began  in  the  summer  of  1990  after  receiving  a 
grant  and  hiring  a  director.  Implementation  of  a  monitoring  and  feedback 
system  and  action  planning  also  occurred  before  the  increase  in  community 
change  was  shown.  The  longest  and  sharpest  climb  in  community  change  was 
shown  beginning  in  fall  1992  after  hiring  of  additional  staff. 
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Figure  10.2 
Kansas  LEAN:  Community  changes  produced 
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Was  There  a  Change  in  Behavior  Related  to  Risks  for 
CVD? 

Behavioral  surveys  can  be  used  to  collect  information  to  address  whether  the 
initiative  produces  a  positive  effect  on  behavior  associated  with  the  risks  for 
CVD.  The  BRFSS,  available  from  the  CDC  in  Atlanta,  is  one  potential  source  of 
available  behavioral  data.  The  survey  includes  questions  about  nutrition,  tobacco 
use,  and  physical  activity  (Chapter  7). 

Figure  10.3  shows  data  from  the  BRFSS  on  the  percentage  of  people  who 
report  daily  or  weekly  use  of  cigarettes  at  two  points  in  time  (1991,  the  first  year 
of  implementation;  1992,  the  second  year)  for  a  hypothetical  CVD  prevention 
project.  Data  from  the  target  county  are  compared  with  statewide  data. 
Reported  regular  use  of  cigarettes  decreased  in  the  target  county  with  a  slight 
increase  statewide.  Although  other  factors  may  account  for  the  results,  these 
findings  suggest  that  the  initiative  may  have  had  a  modest  effect  on  tobacco  use. 
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Figure  10.3 
Reported  daily  or  weekly  cigarette  use  among  adults 


Target  community  Statewide 

■  1991Q1992 


Source:  BRFSS. 


Were  Members  Satisfied  with  the  Partnership? 

Support  and  challenges  for  the  initiative  may  be  suggested  by  ratings  of 
member  satisfaction.  We  recommend  using  a  Member  Satisfaction  Survey  to 
gather  data  on  satisfaction  with  the  initiative  in  a  number  of  areas  such  as 
competence  of  staff  and  leadership.  Each  item  on  the  survey  is  rated  on  a  5- 
point  scale,  with  5  being  the  most  positive  (see  Chapter  5). 

Results  from  Kansas  LEAN  will  illustrate  data  obtained  from  a  satisfaction 
survey.  Strength  and  competence  of  leadership  (average  rating  =  4.4)  and  staff 
(average  rating  =  4.4)  were  among  the  highest  rated  items  by  the  membership 
of  the  initiative.  Participation  by  racial/ethnic  minorities  (average  rating  =  3.3) 
and  promoting  awareness  of  the  goals,  actions,  and  accomplishments  of  the 
initiative  (average  rating  =  3.5)  were  among  the  lower  rated  items.  All 
respondents  noted  that  the  community  was  better  off  because  of  Kansas  LEAN. 
The  data  and  qualitative  comments  from  the  survey  informed  leadership  about 
members'  perceptions  of  the  functioning  of  the  partnership  and  challenges  to  be 
addressed. 
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Were  the  Community  Changes  Important  to  Reducing 
Risks  for  CVD? 

Community  changes  or  accomplishments  of  the  initiative  are  of  unequal 
importance.  We  recommend  using  a  constituent  survey  of  outcomes  to  assess 
the  perceived  significance  of  the  community  changes  facilitated  by  the  initiative 
that  are  related  to  the  mission.  Members  of  the  partnership  and  outside  experts 
rate  each  community  change  using  a  5-point  scale,  with  5  the  highest 
importance  rating  and  1  the  lowest  (see  Chapter  6).  Note  that  data  on 
community-level  indicators  provide  a  more  quantitative  assessment  of  public 
health  significance  (see  Chapter  8). 

Data  from  an  outcome  survey  for  Kansas  LEAN  will  illustrate.  Overall, 
respondents  rated  community  changes  facilitated  by  the  project  as  "Important." 
Among  the  higher  rated  changes  was  a  supermarket  intervention  that  included 
price  reduction,  shelf  prompts,  and  posters  (average  rating  =  4.1).  The 
contribution  of  all  the  reported  community  changes  to  the  mission  was  also 
rated  positively.  The  data  were  shared  with  initiative  leadership  and  used  in 
future  planning.  The  results  were  also  used  to  inform  funders  about  the 
importance  of  the  accomplishments  of  the  state  wide  coalition. 


Does  the  Initiative  Have  a  Community-level  Outcome 
Related  to  the  Risks  for  CVD? 

Archival  records  or  other  indirect,  global  measures  are  used  to  provide  data  on 
whether  the  initiative  is  having  a  positive  impact  on  community-level  indicators 
(see  Chapter  8).  Figure  10.4  provides  hypothetical  data  on  the  seasonally 
adjusted  trend  in  per  capita  consumption  of  cigarettes  (patterned  after  a  similar 
analysis  in  California).  This  figure  shows  that  the  new  State  tax  and  tobacco-free 
coalition  may  have  contributed  to  the  downward  trend  in  tobacco  use,  extending 
the  more  dramatic  changes  that  correlated  with  new  federal  and  State  taxes  on 
tobacco  products.  Data  from  community-level  indicators  may  be  used  to 
demonstrate  the  level  of  the  problem  in  the  county  or  State.  By  tracking  these 
data  over  the  years  of  the  initiative  and  comparing  levels  with  comparable 
counties  or  States  we  can  estimate  the  overall  effect. 
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Figure  10.4 

Trend  in  the  community-level  indicator 
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*Seasonally  adjusted  trend  for  a  hypothetical  tobacco-free  coalition. 


Is  Community-level  Outcome  Related  to  Changes 
Facilitated  by  the  Initiative? 

Data  on  both  community-level  outcome  and  related  patterns  of  community 
change  may  be  integrated  to  examine  possible  relationships.1     For  example, 
outcome  data  on  consumption  of  cigarettes  might  be  overlaid  on  community 
changes  facilitated  by  a  tobacco  control  initiative.  Possible  relationships  between 
community  changes  facilitated  by  the  partnership  and  community-level  outcome 
can  then  be  explored. 

Figure  10.5  displays  data  for  a  hypothetical  coalition  whose  goal  is  to  decrease 
sedentary  behavior.  The  figure  shows  community  changes  related  to  a 
community-level  outcome  of  the  percentage  of  adults  who  are  sedentary.  Note 
that  a  reduction  in  the  percentage  of  adults  who  are  sedentary  occurs  only  after 
a  sufficient  number  of  community  changes  accumulated.  With  a  reduction  in 
community  change  in  the  final  year,  no  further  reductions  in  the  percentage  of 
adults  who  are  sedentary  were  reported  for  this  hypothetical  initiative.  This 
example  illustrates  how  data  from  the  evaluation  system  can  be  used  to  examine 
possible  relationships  between  system  changes  facilitated  by  the  initiative  and 
community-level  outcomes. 
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Figure  10.5 
Community  changes  vs.  sedentary  behavior* 
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*Data  for  a  hypothetical  physical  activity  coalition. 


What  Critical  Events  Seemed  to  Spur  Rates  of  Community 
Change? 

Semistructured  interviews  with  leaders  and  active  members  of  the  partnership 
can  be  used  to  identify  events  that  were  influential  to  the  initiative's  functioning. 
Through  analysis  of  critical  events  many  important  factors  may  be  identified 
such  as  hiring  of  staff,  monitoring  and  feedback,  action  planning,  or  a  transition 
of  leadership  (see  Chapter  9). 

We  can  explore  possible  relationships  between  identified  critical  events  and 
measures  of  community  actions  and  community  changes.  This  can  be  done  by 
indicating  the  timing  of  critical  events  on  graphs  of  community  actions  and 
changes.  Figure  10.2  displays  several  critical  events  identified  by  a  variety  of  key 
informants  who  reported  that  the  events  were  influential  in  the  functioning  of 
Kansas  LEAN.  Increases  in  the  rate  of  community  change  occurred  after 
introduction  of  the  grant  award,  completion  of  action  planning,  and  hiring  of 
additional  staff.  These  factors  may  have  contributed  to  increases  in  community 
change.  This  process  may  help  identify  factors  important  in  the  development  of 
initiative.  It  also  helps  document  the  history  of  the  initiative. 
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Conclusion 

This  chapter  describes  how  to  use  this  comprehensive  evaluation  system  to 
examine  a  variety  of  key  questions  about  the  functioning  of  CVD  prevention 
initiatives.  The  next  chapter  describes  how  to  communicate  information  about 
the  partnership  to  secure  support  from  key  audiences. 


Summary  of  questions  that  can  be  addressed 
with  the  evaluation  system 

Was  the  community  mobilized  to  reduce  risks  for  CVD? 

What  changes  in  the  community  resulted  from  the  initiative? 

Was  there  a  change  in  behavior  related  to  risks  for  CVD? 

Were  members  satisfied  with  the  partnership? 

Were  the  community  changes  important  to  reducing  risks  for  CVD? 

Does  the  initiative  have  a  community-level  outcome  related  to  the  risks  for 
CVD? 

Is  the  community-level  outcome  related  to  changes  facilitated  by  the 
initiative? 

What  critical  events  seemed  to  spur  rates  of  community  change? 
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Communicating  Information  About 
the  Initiative 


Leaders  and  evaluators  of  the  partnership  should  communicate  information 
about  the  progress  and  accomplishments  of  the  initiative  with  audiences  of 
interest.  Local  officials  and  potential  funders  may  be  particularly  important 
audiences.  Presenting  data  on  the  process,  intermediate  outcome,  and  more 
distal  outcomes  of  the  initiative  communicates  to  your  audience  that  the 
initiative  has  specific  goals  and  is  monitoring  progress  toward  achieving  those 
goals.  Data  presentations  also  provide  opportunities  for  local,  State,  and 
national  audiences  to  learn  from  and  advise  the  partnership  about  how  to 
improve. 


Communicating  data:    What  and  why? 


What  is  it? 


•  A  way  to  inform  local,  State,  and  national  audiences  of  the  goals  and 
achievements  of  the  CVD  prevention  effort. 

Why  do  it? 

•  At  the  local  level: 

To  help  raise  awareness  of  CVD  as  an  issue  of  concern 

To  help  attract  volunteers,  funding,  and  in-kind  resources  from  local 
concerned  citizens  and  agencies 

To  promote  recognition  of  the  efforts  of  volunteers  and  collaborators 

To  help  lobby  for  local  ordinances  or  program  changes  to  help  prevent 
CVD 

To  provide  accountability  to  the  community,  trustees,  and  funders. 
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Communicating  data:    What  and  why?  (Continued) 

•  At  the  State  level: 

To  create  a  "name"  for  the  initiative  in  the  State,  making  it  more 
competitive  when  seeking  State  resources 

To  help  establish  a  statewide  network  of  persons  and  agencies  with 
similar  goals 

To  help  lobby  for  legislative  changes  to  help  prevent  CVD 

To  help  the  initiative  garner  recognition  and  resources  from  the  State  and 
region. 

•  At  the  national  level: 

To  create  a  "name"  for  the  initiative  nationwide,  which  makes  it  more 
competitive  when  seeking  resources  from  the  State  or  Federal 
government  or  from  private  foundations 

To  help  tap  into  nationwide  networks  of  persons  and  agencies  interested 
in  preventing  CVD 

To  help  the  initiative  garner  recognition  and  resources  from  across  the 
country 

To  encourage  development  of  community  partnerships  to  prevent  CVD. 

•  At  all  levels: 

To  help  keep  the  issue  of  CVD  prevention  on  the  public  agenda  and 
validate  partnership  strategies  as  appropriate  and  worthy  of  support. 


Accomplishments  to  be  highlighted  in  presentations  will  likely  change  over  time. 
Early  on,  baseline  data  from  behavior  surveys  and  archival  records  of 
community-level  indicators  may  help  create  awareness  of  the  level  of  the 
problem  of  CVD  in  the  community  or  State.  Monitoring  data,  particularly  of 
community  action  and  change,  provide  evidence  of  project  activities  and 
accomplishments  early  in  the  development  of  the  CVD  prevention  initiative. 
Later  in  the  development  of  the  partnership,  behavioral  data,  and  community- 
level  indicators  will  help  document  improvements  in  more  distal  outcomes. 
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Communicating  data:    How  to's 

Develop  a  presentation  format  that  can  be  lengthened  or  shortened 
depending  on  the  amount  of  time  available,  including  compelling 
descriptions  and  visuals  of: 

CVD  and  related  risk  factors  as  issue(s)  of  concern 

The  initiative's  goals,  strategies,  and  methods  for  reaching  its  goals 

Data  on  activities  (e.g.,  services  provided) 

Data  on  accomplishments  (i.e.,  community  changes) 

Data  on  outcomes  (i.e.,  behavioral  measures  and  community-level 
indicators). 

Keep  visuals  simple  to  decrease  problems  interpreting  data;  the  first  and 
last  visuals  and  words  should  convey  your  primary  findings  or  message. 

Identify  important  audiences  for  the  data: 

Local:  Health  organizations,  civic  organizations,  business  groups, 
grassroots  organizations,  school  boards,  PTAs,  church 
organizations,  editors  and  editorial  boards,  newspapers,  elected 
and  appointed  officials  in  local  government,  grantmakers 

State:  State  and  regional  professional  conferences,  regional 
professional  training  workshops,  grassroots  and  advocacy 
organizations,  health  conferences,  grantmakers 

National:  Professional  conferences,  professional  training  workshops, 
grassroots  and  advocacy  organizations,  health  conferences, 
grantmakers. 

Identify  avenues  for  getting  the  word  out: 

Word  of  mouth 
Presentations 
Newspapers 
Newsletters 
Radio  spots 
Television  coverage 
Professional  journals. 
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Data  should  be  graphed  and  shared  with  the  leadership  and  membership  of  the 
partnership.  Presenters  can  share  stories  and  visuals  such  as  photos,  project 
posters,  or  other  materials  from  project  activities.  Combining  data,  visuals,  and 
anecdotes  or  quotes  during  presentations  is  a  powerful  way  to  share  your 
message. 

Presentations  should  be  tailored  to  each  audience  and  their  particular  interests. 
For  instance,  activities  that  contribute  to  health  outcomes  may  be  of  particular 
interest  to  public  health  audiences,  while  service  organizations  may  be  especially 
interested  in  activities  that  contribute  to  community  well-being.  Furthermore, 
who  presents  to  each  audience  may  influence  how  the  audience  receives  the 
message.  For  example,  lay  or  community  members  may  reach  the  audience  the 
best  in  some  settings,  whereas  in  others  academic  or  professional  staff  members 
of  the  initiative  or  those  who  represent  the  business  community  may  be  more 
effective. 

Each  presentation  should  have  a  clear  goal  or  intended  outcome.  Many 
presentations  will  use  data  and  other  relevant  information  to  increase  awareness 
of  CVD  as  a  health  concern  and  the  need  for  action.  It  is  important  to  go 
beyond  increasing  awareness.  Presentations  can  be  used  effectively  to  solicit 
support  and  involvement  of  community  members  in  the  CVD  prevention  effort. 
For  example,  service  organizations  may  volunteer  to  assist  with  project  activities 
or  collaborate  on  a  new  program.  Some  national  organizations  have  local 
chapters  that  support  projects  that  may  be  consistent  with  the  mission  or 
provide  resources  for  special  projects.  Professional  organizations  such  as  the 
American  Heart  Association  and  the  American  Lung  Association  have  expertise 
in  areas  that  may  strengthen  the  initiative.  These  national  organizations  have 
State  and  local  affiliates  that  work  directly  with  the  public  and  are  active  with 
legislators  and  other  policy  makers.  Members  of  State  and  national 
organizations  may  be  able  to  establish  changes  in  policy  or  practice  that  will 
reduce  risks  for  CVD.  For  instance,  local  members  of  the  National  Recreation 
and  Parks  Association  have  been  successful  in  developing  physical  activity 
promotion  materials  that  are  used  in  local  health  departments.  When 
communicating  data,  be  sure  to  have  a  goal  or  outcome  in  mind. 

Publishing  in  professional  journals  is  a  more  formal  and  time-intensive  method 
of  communicating  information  about  the  partnership.  Forming  collaborative 
relationships  with  researchers  at  local  colleges  and  universities  may  increase 
access  to  these  forms  of  communication. 
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Summary  of  possible  goals  of  a  presentation 

Obtaining  money  and  in-kind  resources  for  the  partnership. 

Attracting  volunteers  for  project  activities. 

Influencing  a  change  in  program,  policy,  or  practice. 

Obtaining  input  on  making  the  initiative  more  responsive. 

Overcoming  resistance  to  the  initiative. 

Learning  how  the  initiative  can  become  more  effective. 
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Some  Reflections  on  the 
Evaluation  System 


This  handbook  describes  a  comprehensive  evaluation  system  for  community 
partnerships  for  preventing  CVD.*  We  described  6  measurement  instruments 
and  13  measures  related  to  understanding  and  improving  the  process, 
intermediate  outcome,  and  more  distal  outcomes  of  community  health 
initiatives.  Depending  on  interests,  resources,  and  requirements,  a  community 
partnership  might  use  some  or  all  of  these  measures. 


Core  measurement  instruments 

Monitoring  and  feedback  system. 

Surveys  of: 
Satisfaction 
Outcome 

Behavioral  surveys 
Community-level  indicators 
Interviews  with  key  participants. 


The  data  can  be  used  to  understand  a  variety  of  questions,  including  whether  the 
initiative  is  (a)  mobilizing  the  community,  as  suggested  by  a  steady  climb  in 
community  action;  (b)  changing  the  community,  as  indicated  by  new  or  modified 
programs,  policies,  and  practices  related  to  the  mission;  (c)  changing  behavior, 
as  indicated  by  data  from  behavioral  surveys;  or  (d)  having  more  distal  outcomes, 
as  indicated  by  changes  in  community-level  indicators  of  reduced  risks  for  CVD. 

The  metric  of  community  change  is  an  especially  strong  contribution  of  this 
evaluation  system.  Community  changes  (new  or  modified  programs,  policies,  or 
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*Portions  of  this  chapter  are  adapted  from:  Fawcett,  S.  B.,  Lewis,  R.  K.,  Paine-Andrews,  A., 
Francisco,  V.  T.,  Richter,  K.  P.,  Williams,  E.  L.,  and  Copple,  B.  (in  press). 
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practices)  are  evidence  of  the  initiative's  accomplishments.  High  rates  of 
community  change  can  help  create  a  health-promoting  community  with 
reduced  risks  for  CVD. 

Exploring  the  relationship  between  community  change  and  community-level 
indicators  is  important  because  of  the  long  delay  between  community  changes 
and  positive  changes  in  community-level  indicators.  The  research  with  Project 
Freedom  of  Wichita,  Kansas,  suggests  that  community  change  and  community- 
level  indicators  may  be  positively  related.1  Future  research  is  needed,  however, 
to  improve  understanding  of  the  possible  relationship  between  community 
change  and  community-level  indicators  associated  with  CVD. 

The  evaluation  system  also  provides  qualitative  data  on  how  community 
initiatives  function.  Data  from  surveys  assess  member  satisfaction  with  the 
initiative.  Interviews  with  key  informants  help  determine  what  critical  events, 
such  as  action  planning  or  the  hiring  of  staff,  help  community  health  initiatives 
function  more  effectively. 


Challenges  to  evaluating  community  initiatives 
to  reduce  risks  for  CVD 

Effects  are  often  delayed. 

Difficult  to  establish  links  between  project  activities  and  particular 
outcomes. 

May  not  find  a  suitable  comparison. 

Finding  an  early  marker  of  more  distal  outcomes. 


Evaluating  community  initiatives  for  preventing  CVD  is  challenging  for  a 
number  of  reasons.  First,  the  positive  effects  of  community  initiatives  are  often 
not  shown  for  many  years  and  may  be  difficult  to  track.  Facilitating  positive 
change  in  risk  and  protective  factors  is  likely  to  be  shown  earlier  in  the 
development  of  community  initiatives.  The  evaluation  should  focus  on 
identifying  community  changes  that  will  likely  reduce  risks  (or  enhance 
protection)  for  CVD.  The  various  community  changes  will  not  be  of  equal 
importance.  Surveys  and  interviews  with  key  informants  should  be  used  to 
secure  information  about  the  relative  importance  of  changes  to  the  mission  of 
preventing  CVD.  The  evaluation  should  remain  in  place  long  enough  to 
determine  the  size  and  durability  of  changes  in  community-level  indicators. 
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Second,  determining  the  relationship  between  a  project's  activities  (the 
independent  variable)  and  its  effects  (the  dependent  variables)  is  a  common 
research  goal.  This  may  be  especially  challenging  when  evaluating  community 
health  initiatives.  Community  initiatives  use  a  variety  of  strategies,  such  as 
advocacy  or  media,  to  put  in  place  an  array  of  interventions,  such  as  point-of- 
purchase  prompts  for  low-fat  foods.  The  interventions  target  different  groups, 
such  as  food  professionals  and  parents,  and  are  implemented  by  different 
agents,  such  as  religious  leaders  and  business  people.  They  operate  in  multiple 
community  sectors,  such  as  in  schools  and  businesses,  and  over  different  lengths 
of  time.  Consequently,  clearly  identifying  the  independent  variable,  its  timing, 
and  dose  is  difficult.  The  measure  of  community  change  provides  an  important 
and  flexible  measure  of  environmental  or  systems  change. 

Third,  suitable  comparison  groups  are  difficult  to  secure,  however,  because 
changes  produced  by  the  partnership  often  affect  the  entire  population. 
Obtaining  data  from  (or  even  finding)  similar  communities  may  be  difficult. 
Furthermore,  because  many  communities  may  also  be  addressing  prevention  of 
CVD,  they  may  serve  only  as  a  comparison,  but  not  as  an  experimental  control 
group.  Finding  a  suitable  comparison,  although  difficult,  helps  to  conclude  that 
the  results  were  due  to  the  initiative  and  not  something  else. 

Last,  identifying  early  markers  of  community-level  outcomes  would  allow  early 
and  ongoing  feedback  on  the  progress  of  the  initiative.  As  described  earlier, 
community  change  may  predict  community-level  outcome.  However, 
replication  of  the  possible  relationship  between  community  change  and 
community-level  indicators  is  needed.  Future  longitudinal  studies  of  multiple  and 
diverse  initiatives  may  help  determine  the  conditions  under  which  community 
change  predicts  community-level  outcome. 

The  monitoring  system  addresses  some  of  these  issues.  However,  there  are  also 
methodological  challenges  related  to  accuracy  and  sensitivity  of  reports, 
measurement  reactivity,  and  instrumentation  change.  These  challenges  may  be 
addressed  in  several  ways.  First,  Event  Logs  and  follow-up  interviews  with 
members  and  staff  may  increase  the  accuracy  and  completeness  of  reports. 
Second,  honest  reporting  may  be  encouraged  by  occasional  verification  by 
independent  sources,  such  as  meeting  minutes  and  newspaper  articles.  Third,  a 
more  sensitive  reporting  system  would  also  note  when  events  are  terminated, 
not  only  when  started.  Fourth,  there  is  likely  to  be  a  high  level  of  reactivity  to 
the  monitoring  system.  In  fact,  the  system  was  designed  to  facilitate  reactivity. 
Because  data  collection  and  feedback  are  part  of  the  design  of  partnerships 
using  this  evaluation  system,  the  observed  effects  may  be  limited  to  those 
partnerships  using  monitoring  and  feedback.  Fifth,  the  evaluation  can  be 
expensive  and  time  consuming.  Users  are  encouraged  to  modify  the  evaluation 
(see  the  "Evaluation  on  a  Shoestring  Budget"  boxes  throughout  this  handbook). 
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Finally,  changes  in  instrumentation  can  be  minimized  by  using  the  expanded 
scoring  instructions  and  behavioral  definitions  found  in  Chapter  4  and  Appendix 
A.  High  levels  of  agreement  in  scoring  among  independent  observers  indicates 
that  the  data  collected  by  the  monitoring  system  can  be  reliable. 


Strategies  for  evaluating  community  initiatives 

Focus  the  evaluation  on  community  change,  a  promising  early  marker  of 
impact. 

Provide  ongoing  feedback. 

Assess  the  significance  of  changes. 

Ensure  data  quality  through  verification  and  reliability  assessments. 

Examine  community-level  indicators  of  more  distal  outcomes. 

Use  multiple  measures,  including  quantitative  and  qualitative  data. 

Use  qualitative  data  to  examine  critical  events. 

Replicate  the  findings  in  multiple  case  studies. 

Conduct  the  evaluation  long  enough  to  learn  about  the  size  and  durability 
of  outcomes. 


Even  with  these  challenges,  the  study  of  initiatives  will  likely  continue  to  increase 
understanding  of  the  development  and  functioning  of  community  health 
initiatives.  Versions  of  this  evaluation  system  have  been  used  by  researchers 
from  the  University  of  Kansas  Work  Group  on  Health  Promotion  and 
Community  Development  with  health  and  human  service  initiatives  in 
Massachusetts  funded  by  the  W.  K.  Kellogg  Foundation  and  with  the  Decade  of 
Hope  Coalition  in  Dulce,  New  Mexico,  supported  by  the  Center  for  Substance 
Abuse  Prevention's  Community  Partnership  Program.  With  support  from  the 
Kansas  Health  Foundation,  there  are  11  sites  involved  in  a  replication  of  a 
substance-abuse  prevention  model,  a  school  and  community  initiative  for  the 
prevention  of  adolescent  pregnancy,  and  a  rural  health  promotion  initiative. 
These  multiple  case  studies  provide  important  opportunities  for  exploring  how 
and  under  what  conditions  community  initiatives  effect  community  change  and 
community-level  outcome.  This  work  and  future  efforts  will  test  the  strength  of 
this  model  for  evaluating  community  initiatives  and  its  usefulness  for  supporting 
community  partnerships  to  reduce  risks  of  CVD. 
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Few  research  studies  exist  that  show  that  an  investment  in  community 
partnerships  had  a  measurable  benefit  on  community  health.  Participatory 
evaluation  approaches  such  as  those  illustrated  by  this  evaluation  system  help  us 
understand  the  factors  associated  with  the  functioning  of  community  health 
initiatives.  This  information  can  be  used  to  build  and  strengthen  community 
capacities  to  produce  change.  Collaborative  research  with  community  initiatives 
contributes  to  societal  change  and  the  science  that  may  serve  as  its  foundation. 


Summary  lessons  on  evaluation  of  collaborative  partnerships 

The  primary  purpose  of  the  evaluation  is  to  support  improvement,  not  to 
judge  success  or  failure. 

Evaluation  information  can  empower  community  partnerships  to  further 
develop  and  renew  themselves. 

Evaluation  should  begin  early  and  be  an  integral  part  of  the  development 
process. 

Evaluation  should  be  a  participatory  and  collaborative  process. 

A  monitoring  system  can  help  community  leadership  establish  and 
maintain  effective  functioning  of  the  initiative. 

Feedback  should  be  provided  at  regular  intervals,  especially  early  in  the 
initiative's  development. 

Evaluation  information  helps  to  discover  whether  the  partnership's  efforts 
are  actually  effective. 

Evaluation  information  helps  direct  the  initiative's  attention  to  powerful 
variables  that  might  actually  make  a  difference. 

Evaluation  results  remind  us  that  behavior  change  is  often  slow  and 
dependent  on  multiple  changes  in  the  environment. 

Evaluation  results  should  be  communicated  openly  and  frequently  to  all 
leadership,  membership,  trustees,  and  funding  agents. 

Grantmakers  can  use  evaluation  information  to  encourage  productivity 
and  accountability. 

Community  leadership  can  use  evaluation  information  to  attract  and 
maintain  support  and  resources. 
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Appendices 


Appendix  A 


Background  Information 
and  Materials  for  Using  the 
Monitoring  System 


Chapter  4  introduced  the  monitoring  system  and  outlined  the  five  major  steps 
involved  in  completing  the  monitoring  process.  Appendix  A  provides  more  detailed 
information  for  people  responsible  for  completing  the  three  most  complex  steps:  cat- 
egorizing log  entries,  assessing  reliability,  and  graphing  and  providing  feedback  on 
the  data  generated  by  the  monitoring  system. 


Categorizing  Log  Entries 


This  section  addresses: 

•  General  scoring  instructions. 

•  Multiple  events  in  one  log  entry. 

•  Duplicate  scores. 

•  Relationship  between  community  action  and  community  change. 

•  Verifying  reported  events. 

•  Recordkeeping. 

•  Expanded  definitions  of  categories  for  scoring. 

•  Practice  exercises  and  answers. 
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General  Scoring  Instructions 

This  section  provides  general  guidance  for  deciding  which  events  fit  into  which 
scoring  category.   Categorizing  events  is  not  an  exact  science.  As  evaluators 
gain  experience  classifying  events,  they  may  want  to  make  additions  to  clarify 
the  definitions  used  to  categorize  events.    Adding  examples  of  events  that  are 
difficult  to  categorize  will  help  others  using  the  monitoring  system. 

Table  A.  1  provides  a  summary  of  the  observational  code  used  to  categorize 
events.  It  provides  a  list  of  categories  of  key  events,  definitions,  and  examples. 
What  is  an  event?  Broadly,  events  are  occurrences  that  are  designed  to  reduce 
risks  for  CVD.  The  distinction  between  external  and  internal  events  will  assist  in 
categorizing  key  events. 


Table  A.1 

Summary  of  the  observational  codes 

for  the  monitoring  system 


Code 


Definition 


Examples 


External  Events  (Happens  outside  the  initiative) 

Actions  taken  in  the  community  to  bring 


CA 

Community 
Actions 

CC 

Community 
Changes 


about  a  new  or  modified  program,  policy, 
or  practice  to  reduce  risks  for  CVD 

New  or  modified  programs,  policies,  or 
practices  in  the  community  facilitated  by  t 
he  initiative  that  reduces  risks  for  CVD 


Letters 
Phone  calls 
Rally 

A  new  (or  modified) 
program  (e.g.,  nutrition 
education) 

A  new  (or  modified)  policy 
(e.g.,  labeling  low-fat  foods) 
A  new  (or  modified)  prac- 
tice (e.g.,  regarding  hours 
of  service) 


SP 

Service 
Provided 

Events  that  are  designed  to  provide 
information,  instruction,  or  develop 
skills  of  people  in  the  community 

•  Classes 

•  Workshops 

•  Communications  such  as 
bill  stuffers 

M 

Media 

Coverage 

Coverage  of  the  initiative  or  its  projects 
in  the  newspaper,  radio  or  television,  or 
newsletter 

•  Radio 

•  TV  (e.g.,  PSAs) 

•  Brochure 

X 
Other 

Items  for  which  no  code  or  definition  has 
been  created 
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Table  A.1 

Summary  of  the  observational  codes 

for  the  monitoring  system  (Continued) 


Internal  Events  (Happens  within  the  initiative) 


PP 

Planning 
Products 


Results,  or  products  of  planning  activities 
within  the  group 


RG  Acquisition  of  funding  for  the  initiative 

Resources         through  grants,  donations,  or  gifts  in-kind 
Generated 


Statements  of  objectives 

Formation  of  committees  or 

task  forces  with  established 

members 

Grant  application 

Materials  received 
People's  time  (except 
volunteers) 
Money 


External  Events.  Most  of  the  events  will  involve  people  not  directly  associated 
with  the  initiative.  For  example,  staff  may  work  with  the  city  council  to  approve 
a  new  ordinance  to  ban  smoking  or  may  co-sponsor  a  religious  organization's 
walking  group.  Both  of  these  events  include  people  from  outside  the  initiative 
(the  city  council  and  walking  group  organizers  and  participants)  and  are 
considered  external  events.  External  events  can  be  classified  as  community 
actions,  community  changes,  services  provided,  or  media  coverage.  External 
events  involve  making  things  happen  in  the  community  to  reduce  the  risks  for 
CVD,  which  is  the  purpose  of  the  initiative. 

Internal  Events.  Some  events  facilitate  the  development  of  the  partnership  or 
coalition.  These  events  may  be  internal,  involving  only  those  working  directly 
with  the  group.   For  example,  the  Steering  Committee  may  complete  their 
strategic  planning  process  and  adopt  a  formal  action  plan,  or  an  executive  from 
the  initiative's  Board  of  Directors  may  donate  office  supplies.  Planning 
products  (such  as  the  first  example),  by  definition,  are  always  internal  events. 
Resources  generated  (e.g.,  volunteer's  professional  time,  donated  materials,  or 
money)  are  internal  events  if  the  beneficiary  is  the  partnership. 

The  distinction  between  external  and  internal  events  is  helpful,  especially  in 
deciding  whether  the  events  should  be  categorized  at  all.  For  example,  if  talk 
about  a  proposed  policy  change  occurred  only  among  members  of  the  initiative, 
not  with  those  who  could  enact  or  implement  the  policy,  it  would  be  considered 
a  planning  activity,  not  a  community  action.   If  the  event  involved  only  those 
from  within  the  initiative,  acting  as  initiative  members,  then,  to  be  categorized, 
the  event  would  need  to  result  in  a  planning  product  or  generate  resources. 
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Multiple  Events  in  One  Log  Entry 

A  single  reported  entry  may  contain  several  discrete  events  that  should  be 
reflected  in  the  scoring  by  being  recorded  separately.  This  is  done  best  by 
breaking  out  the  one  entry  into  several  items  and  scoring  each  event  separately. 
For  example,  the  following  entry  might  be  recorded  on  a  log  form:  "The  second 
block  party  was  facilitated  in  the  Pinkney  neighborhood.   Low-fat  foods  and 
recipes  were  distributed  and  line  dance  lessons  were  given.  The  event  was 
filmed  by  the  local  TV  station  and  appeared  in  the  evening  news."  The  event  is 
a  service  provided,  and  the  section  reporting  media  coverage  would  be  scored 
media. 

Duplicate  Scores 

There  is  meant  to  be  little  overlap  between  definitions.   For  instance,  an  item 
may  not  be  scored  as  both  a  planning  product  and  service  provided.  The 
category  of  community  change  is  an  exception:   items  scored  as  community 
change  are  usually  accompanied  by  an  additional  scoring  category.  The  intent 
of  the  category  of  community  change  is  to  document  all  instances  of  new  or 
modified  programs,  policies,  and  practices.   But,  a  new  program  (such  as  a 
workshop)  is  also  scored  as  a  service  provided  {see  Table  A.  1).  To  capture  all  of 
the  changes  produced,  the  first  new  program  and  the  first  new  practice  are 
scored  as  community  changes  in  addition  to  another  category  {resources 
generated,  community  action,  media  coverage,  or  services  provided). 

The  most  likely  combinations  are  community  change  and  service  provision 
(e.g.,  the  first  instance  of  risk  assessments  conducted  at  a  grocery  store  chain 
may  be  scored  as  both).  Community  change  and  community  action  may  also 
be  reported  as  a  combined  item  (e.g.,  "Committee  member  met  with  the 
hospital  director.  Routine  heart  health  screenings  for  family  members  will  now 
be  offered  in  the  waiting  rooms").   Only  events  scored  as  community  change 
can  have  more  than  one  score. 

Relationship  Between  Community  Action  and  Community  Change 

Community  actions  and  community  changes  generally  relate  to  each  other. 
Keep  in  mind  the  goal  or  outcome  of  an  action  when  scoring  it.  The  purpose  of 
a  community  action  is  to  make  some  change  in  program,  policy,  or  practice 
designed  to  reduce  risks  for  CVD  (a  community  change).    For  each 
community  action,  the  intended  community  change  should  be  evident.  A 
person  filling  out  a  log  may  word  items  to  fit  a  particular  category  or  definition. 
The  e valuator  must  code  the  item  relative  to  what  actually  happened. 
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Verifying  Reported  Events 

If  the  person  who  scores  the  logs  is  different  from  the  person  who  fills  them  out, 
it  will  be  important  to  spot-check  information  on  the  log  forms  for  accuracy. 
The  focus  of  verification  should  be  on  items  categorized  as  community 
changes,  because  this  may  be  a  particularly  important  marker  of  program 
impact. 

One  way  to  verify  events  is  to  choose  several  recorded  items  and  call  the 
person(s)  involved.  You  may  need  to  get  the  phone  number  from  the  person 
completing  the  logs.  If  an  item  that  is  selected  for  verification  cannot  be  verified, 
do  not  include  it  when  counting  items  in  each  scoring  category.  Caution:  this  is 
a  highly  intrusive  method  of  verification  and  may  suggest  mistrust  to 
collaborators. 

A  less  intrusive  way  of  verifying  key  events  is  to  simply  review  the  logs  and  check 
off  the  events  already  heard  about  from  independent  sources.  Another 
recommended  technique  is  to  examine  archival  records  and  written  evidence 
such  as  meeting  minutes,  newspapers,  and  newsletters  that  may  be  included 
with  the  logs. 

Record  Keeping 

There  are  many  ways  to  archive  data  collected  using  the  monitoring  system. 
These  include  storing  the  logs  in  a  file  cabinet,  saving  logs  typed  into  a  word- 
processing  program,  or  maintaining  a  computerized  database.  Keeping  records 
organized  is  very  important.   Graphs  of  the  data  are  only  summaries;  actual 
entries  of  events  provides  more  detailed  information. 

Entries  in  the  logs  should  be  as  complete  as  possible,  especially  events 
categorized  as  community  changes.  One  rule  of  thumb  is  to  archive 
community  changes  clearly  enough  so  that  someone  else  reviewing  the  same 
entry  (years  later)  would  also  categorize  it  as  a  community  change.  Entries  of 
community  changes  are  used  in  other  parts  of  the  evaluation  system  (see 
Chapter  6,  the  Survey  of  Outcomes).  In  addition,  the  list  of  community  changes 
provides  a  nice  history  of  the  group's  accomplishments. 

Expanded  Definition  of  Scoring  Categories 

In  the  following  section,  each  scoring  category  summarized  in  Table  A.l  is 
defined  in  detail.   Entries  on  monitoring  log  forms  are  compared  to  these 
definitions  to  determine  which  category  best  describes  the  entry.   For  each 
scoring  category,  the  category  is  named  and  abbreviated,  a  general  definition  is 
given,  and  the  criteria  that  log  entries  must  meet  to  be  included  in  the  category 
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are  presented.  The  section  is  outlined  using  a  numbering  system  to  facilitate 
referencing  specific  parts  of  the  definition. 

1 .    Community  Actions  —  Abbreviation:  CA. 

General  definition:    Community  actions  are  actions  taken  to  bring  about  a  new 
or  modified  program,  policy,  or  practice  to  reduce  risks  for  CVD.   Events 
categorized  as  community  actions  document  the  extensive  effort  it  takes  to 
make  change  in  the  community. 

Project  members  performing  community  actions  may  (a)  attempt  to  modify  or 
create  a  new  program,  policy,  or  practice  within  the  community;  (b)  implement 
plans  for  addressing  issues  in  the  community;  or  (c)  attend  meetings  involving 
community  leaders  in  which  changes  are  recommended  by  the  member  of  the 
initiative. 

Community  actions  include  acting  directly  to  make  changes  in  the  community, 
actively  lobbying,  or  advocating  with  change  agents.  Personal  contacts,  phone 
calls,  demonstrations,  petitions,  and  letter  writing  are  all  examples  of 
community  actions. 

1.1.  Community  actions  must  meet  all  of  the  following  criteria: 

1.1.1.  occurred  (not  just  planned); 

1.1.2.  include  community  members  external  to  the  initiative  or  out- 
side the  committee  or  subcommittee  advocating  for  change; 

1.1.3.  be  taken  to  bring  about  changes  in  programs,  policies,  or 
practices  in  the  community;  and 

1.1.4.  be  related  to  the  initiative's  goals  and  objectives. 

1.2.  If  presentations  to  community  audiences  include  generating  changes  to 
be  made  in  the  community  (e.g.,  listening  sessions)  or  are  aimed  specif- 
ically at  some  change  in  the  community  (relative  to  the  group's  mis- 
sion), then  it  is  a  community  action.   If  not,  a  workshop  or  other 
presentation  is  scored  as  a  service  provided. 

1.3.  Collaboration  with  community  members  (people  external  to  the  initia- 
tive) to  set  new  agendas  for  the  community  are  community  actions.   If 
this  is  the  first  occurrence  of  collaboration  in  the  community,  however, 
it  could  be  a  community  change  (a  change  in  practice)  as  well  as  a 
community  action. 

1.4.  Actions  taken  to  keep  the  group  going — working  on  by-laws,  soliciting 
funding  for  the  group,  or  holding  meetings  among  group  members — 
are  not  considered  to  be  community  actions,  since  they  do  not  contrib- 
ute directly  to  changes  in  the  community  to  reduce  CVD.   Internal 
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meetings  among  group  members  are  usually  not  considered  commu- 
nity actions. 

1.4.1.     Exceptions  occur  when  members  of  groups  targeted  for 

change  are  also  involved  in  the  initiative  and  its  committees 
and  task  forces.  For  example,  at  a  committee  meeting, 
enforcement  issues  for  selling  tobacco  products  to  minors 
might  be  discussed  with  a  representative  of  the  police  depart- 
ment. Because  a  representative  of  a  community  sector  to  be 
changed  (i.e.,  law  enforcement)  was  involved,  it  would  be  con- 
sidered a  community  action. 

Examples  of  Community  Actions: 

•  A  staff  member  discussed  menu  changes  and  offered  assistance  in 
developing  a  heart  healthy  menu  at  a  local  restaurant.  (Community  action 
because  it  facilitates  a  practice/policy  change.  See  definition  1.1.) 

•  Merchants  were  asked  to  display  signs  describing  the  penalty  for  selling 
tobacco  to  minors  and  the  need  for  proper  identification.   (Community 
action  because  it  is  directly  related  to  a  community  change  relevant  to  the 
mission  of  reducing  risks  for  CVD.   See  definition  1.1.) 

•  Phone  calls  were  made  to  managers  of  three  area  supermarkets  to  discuss 
the  possibility  of  placing  point-of-purchase  signs  on  shelves  indicating  items 
low  in  saturated  fat.   (Community  action  because  it  is  directly  related  to  a 
community  change  relevant  to  the  mission  of  reducing  risks  for  CVD.   See 
definition  1.1.) 

•  A  town  meeting  was  held  with  residents  of  the  Pinkney  neighborhood  to 
discuss  how  to  increase  opportunities  for  physical  activity.  Ideas  generated 
were  added  to  the  project's  action  plan.  (Action  taken  to  generate  ideas  for 
community  change.   See  definition  1.2.) 

Examples  of  Items  not  Scored  as  Community  Actions: 

•  A  meeting  was  held  by  the  subcommittee  for  public  policy  to  discuss 
community  policies  which  may  be  related  to  welfare  reform.   (This  is  not  a 
community  action,  since  no  one  external  to  the  initiative,  such  as  a 
policymaker,  was  present  and  it  was  not  part  of  the  mission  of  the  initiative. 
See  definitions  1.1.2,  1.4,  and  1.1.4.  This  entry  would  be  scored  "X."    X  is 
the  abbreviation  for  the  category  "Other"  for  items  for  which  no  code  or 
definition  has  been  created). 
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The  executive  director  contacted  area  store  managers  to  arrange  a  meeting 
to  discuss  the  goals  of  the  initiative  and  request  their  support.  (This  is  not  a 
community  action,  since  the  aim  was  to  increase  support  for  the  initiative, 
not  community  change.  See  definitions  1.1.3  and  1.4.  This  entry  would  be 
scored  X.) 

A  meeting  was  held  by  the  Schools  Committee  to  discuss  election 
procedures  for  electing  the  chairperson.   (This  is  not  a  community  action, 
since  it  relates  to  change  in  the  initiative,  not  the  community.   See 
definitions  1.1.2  and  1.4.   This  entry  would  be  scored  X.) 

Representatives  of  the  initiative  will  contact  the  local  supermarket  to  arrange 
a  meeting  to  discuss  the  implementing  point-of -purchase  signs.  (This  item  is 
a  future  event,  not  an  action  that  already  occurred.   See  definition  1.1.1. 
This  entry  would  be  scored  X.) 


2.    Community  Changes  —  Abbreviation:  CC. 

General  definition:   Community  changes  are  new  or  modified  programs, 
policies,  or  practices  in  the  community  facilitated  by  the  initiative  that  reduce 
risks  for  CVD.  Statements  of  community  changes  should  include  information 
about  the  impact  on  the  community  (e.g.,  number  of  stores  changing  policies). 
Changes  that  have  not  yet  occurred,  those  unrelated  to  the  group's  goals,  or 
those  which  the  initiative  had  no  role  in  facilitating  are  not  considered 
community  changes  for  the  initiative. 

2.1.  Community  changes  must  meet  all  of  the  following  criteria: 

2.1.1.  occurred  (not  just  planned) ; 

2.1.2.  include  community  members  external  to  the  initiative  or  out- 
side the  committee  or  subcommittee  advocating  for  change; 

2.1.3.  are  related  to  the  initiative's  chosen  goals  and  specific  objec- 
tives; 

2.1.4.  are  new  or  modified  programs,  policies,  or  practices  of  gov- 
ernmental bodies,  agencies,  businesses,  and  other  sectors  of 
the  community;  and 

2.1.5.  are  facilitated  by  individuals  who  are  members  of  the  initiative 
or  are  acting  on  behalf  of  the  initiative. 

2.2.  Changes  reported  at  different  points  in  time  should  be  counted  as  sep- 
arate changes  only  if  they  resulted  from  different  actions. 
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2.3.  The  first  instance  of  a  new  program  or  practice  in  the  community  is 
scored  as  a  community  change,  since  it  constitutes  a  change  in  a  pro- 
gram or  practice  in  the  community. 

2.4.  The  first  occurrence  of  collaboration  between  community  members 
external  to  the  initiative  is  a  community  change  (a  change  in  practice). 

2.5.  Not  all  first-time  events  are  community  changes;  the  event  must  meet 
all  parts  of  the  definition  of  a  community  change.    For  example,  if  staff 
members  attended  a  seminar  for  the  first  time,  this  is  not  a  community 
change  because  it  is  not  a  new  or  modified  program,  policy,  or  prac- 
tice of  an  organization. 

Examples  of  Community  Changes: 

•  Point-of -purchase  signs  indicating  breakfast  cereals  low  in  saturated  fat  were 
put  in  three  supermarkets.  (Change  in  practice  directly  related  to  group 
actions  and  consistent  with  its  mission  of  reducing  risks  for  cardiovascular 
disease.  See  definition  2.1.) 

•  A  video  describing  healthy  nutrition  and  exercise  is  now  shown  daily  in  the 
waiting  room  of  the  pediatric  clinic  of  the  local  hospital.  (Change  in  policy 
directly  related  to  group  actions.  See  definition  2.1.) 

•  A  local  business  adopted  a  low-fat  menu  in  their  employee  cafeteria. 
(Change  in  practice  directly  related  to  actions  by  the  group.  See  definition 
2.1.)  ' 

•  An  education  and  skills  training  program  was  incorporated  into  the  local 
high  school  curriculum.  (Change  in  program  directly  related  to  the  group's 
actions  and  specific  objectives.  See  definition  2.1.) 

Examples  of  Items  Not  Scored  as  Community)  Changes: 

•  Junior  high  school  students  will  increase  awareness  of  the  effects  of  diet  and 
exercise  on  their  physical  and  emotional  health.   (Outcome  written  in  the 
future  tense.  It  will  only  be  scored  if  it  already  occurred.  See  definition 
2.1.1.  This  entry  would  be  scored  X.) 

•  A  new  subcommittee  was  formed  to  address  federal  legislative  issues.  (This 
is  a  planning  product,  since  it  reports  a  change  in  the  organization  of  the 
initiative,  not  the  community.  See  definitions  2.1.2,  2.1.3,  and  2.1.4.  The 
score  for  this  entry  is  X.) 

•  The  project's  Administrative  Assistant  reported  that  the  AME  church  started 
a  new  Sunday  afternoon  walking  group.  (As  written,  the  event  was  not 
facilitated  by  the  project.  See  definition  2.1.5.  The  entry  would  be  scored  X.) 
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3.    Planning  Products  —  Abbreviation:  PP. 

General  definition:   Planning  products  refer  to  the  results  or  products  of 
planning  activities  within  the  group.  There  are  many  types  of  planning 
activities,  such  as  developing  a  mission,  completing  a  strategic  planning 
process,  developing  an  action  plan,  and  setting  committee  goals.  Usually  there 
is  some  result  of  planning,  something  that  helps  guide  the  initiative's  activities. 
The  result  of  planning  can  be,  for  example,  mission  statements,  strategic  plans, 
written  action  plans,  or  written  committee  goals.  These  results  or  products  of 
planning  are  categorized  as  planning  products.  A  new  initiative  will  usually 
complete  a  number  of  planning  products  over  time.  Most  initiatives  review  and 
update  their  action  plans  yearly. 

Planning  products  can  include  (a)  statements  of  objectives  (including  broad 
goals),  (b)  formation  of  committee  or  task  forces  (among  established  members  of 
the  initiative),  (c)  by-laws  and  rules  adopted,  (d)  grant  applications  written  or 
submitted,  and  (e)  the  hiring  of  staff  for  the  initiative. 

3.1.  Planning  products  must  meet  all  of  the  following  criteria: 

3.1.1.  are  identified  products  or  residuals  of  planning  activities, 

3.1.2.  are  completed,  and 

3.1.3.  occurred  with  only  individuals  internal  to  the  initiative. 

3.2.  Planning  products  may  create  the  opportunity  for  service  delivery, 
gathering  and  distributing  resources,  and  community  actions  and  com- 
munity changes. 

3.3.  Separate  planning  products  may  be  scored  if  the  same  item  reflects 
more  than  one  product  of  planning. 

3.4.  Planning  is  an  internal  activity.   Collaboration  with  community  mem- 
bers (people  external  to  the  initiative)  to  set  new  agendas  for  the  com- 
munity are  community  actions.   One  needs  to  keep  in  mind  that 
partners  often  wear  two  or  more  hats  and  can  act  within  the  initiative 
as  well  as  outside  the  initiative. 

3.5.  Planning  products  include  creation  of  groups  within  the  initiative,  such 
as  committees  or  task  forces  for  collaborative  problem  solving. 

3.6.  Hiring  staff  is  an  instance  of  a  planning  product. 

3.7.  Adopting  mission,  objectives,  action  plans,  bylaws,  or  rules  are  plan- 
ning products. 

3.8.  Events  that  lead  to  and  support  the  resulting  planning  product  (such  as 
planning  meeting)  are  scored  as  X. 


140  CVD  Evaluation  Handbook 


Examples  of  Planning  Products: 

•  Legislative  committee  established  and  members  are  elected.   (Committee 
formation  is  a  planning  product.   See  definition  3.5.) 

•  By-laws  were  formally  adopted  by  the  group.  (Adoption  of  by-laws  or  rules 
is  a  planning  product.  See  definition  3.7.) 

•  Community  changes  to  be  sought  by  the  supermarket  task  force  were 
adopted.  The  first  committee  meeting  was  on  6/2/90.   (Adopting 
objectives  is  a  planning  product.  See  definition  3.7.) 

•  Broad  goals  and  objectives  of  the  initiative  were  established  at  initiative 
meeting  on  7/21/90.   (Goal  formation  is  a  planning  product.  See 
definition  3.7.) 

Examples  of  Items  That  are  Not  Planning  Products: 

•  The  initiative  director  drafted  goals  for  the  members  to  review.  (This  is  not  a 
planning  product,  since  the  result  is  yet  to  be  reported.  See  definition  3.1.2. 
Entry  would  be  scored  as  X.) 

•  A  local  foundation  provided  a  $10,000  grant  to  facilitate  an  education  and 
awareness  program  of  the  initiative.   (The  grant  is  a  unit  of  resources 
generated;  the  grant  application  would  likely  be  a  planning  product.   See 
definitions  3.1  and  6.1.) 

•  Travel  arrangements  were  made  for  speakers  to  present  at  the  October 
workshop.  (This  is  not  a  planning  product,  since  it  relates  to  a  future  service 
provided.  See  definition  3.1.  Entry  would  be  scored  as  X.) 

•  Nutrition  education  workshops  were  conducted  with  local  child  care 
providers.   (This  is  a  service  provided.  See  definitions  3.1  and  4.1.) 


4.    Services  Provided  —  Abbreviation:  SP. 

General  definition:  Services  provided  are  events  that  are  designed  to  provide 
information  or  instruction  or  to  develop  skills  of  people  in  the  community. 
Services  provided  include  classes,  programs,  screenings,  workshops, 
publications,  and  other  services  or  communications  (e.g.,  PSAs  and  bill 
stuffers).    Records  on  services  provided  include  the  number  of  classes  or 
programs  conducted  and  the  number  of  participants  in  those  classes/programs. 
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4.1.  Services  provided  must  meet  all  of  the  following  criteria: 

4.1.1.  are  services  or  communications  to  educate,  inform,  enhance 
skills,  or  provide  support; 

4.1.2.  are  sponsored  or  facilitated  by  members  of  the  initiative; 

4.1.3.  occurred  and/or  are  ongoing;  and 

4.1.4.  be  delivered  to  community  members  outside  of  the  initiative. 

4.2.  When  a  new  program  is  initiated,  it  should  be  coded  as  both  a  commu- 
nity change  and  a  service  provided.  Any  continuing  instance  of  pro- 
grams are  services  provided. 

4.3.  If  presentations  to  community  audiences  include  generating  changes  to 
be  made  in  the  community  (e.g.,  listening  sessions)  or  are  aimed  specif- 
ically at  some  change  in  the  community  (relative  to  the  group's  mis- 
sion), then  it  is  a  community  action.   If  not,  a  workshop  or  other 
presentation  is  scored  as  a  service  provided. 

4.4.  Instances  of  service  provision  (e.g.,  each  workshop,  class,  or  program) 
are  scored  each  time  the  event  occurs. 

4.5.  Events  to  plan  services  (such  as  meetings  to  decide  the  content  of  a 
class)  are  scored  as  X. 

Examples  of  Services  Provided: 

•  A  press  conference  on  nutrition  was  held  at  Dillon's  Grocery  Store  #65  and 
attended  by  approximately  100  people.  (This  is  a  service  provided,  since  it 
provided  an  educational  opportunity  related  to  the  project's  mission.   See 
definitions  4.1  and  4.3.) 

•  A  skills  training  workshop  was  conducted  at  the  school.   (This  is  a  service 
provided,  since  it  is  a  workshop  related  to  reducing  risks  for  CVD.   See 
definitions  4.1  and  4.4.) 

•  Nutrition  education  workshops  were  conducted  by  child  care  providers. 
(This  is  a  service  provided,  since  it  is  a  workshop  related  to  reducing  risks  for 
CVD.   See  definitions  4.1  and  4.4.) 


• 


A  conference  on  diet  and  exercise  programs  in  area  businesses  was 
conducted  on  6/27/90.   (This  is  a  service  provided,  since  it  is  an 
educational  program  related  to  reducing  risks  for  CVD.  See  definitions  4.1 
and  4.4.) 
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Examples  of  Items  Not  Scored  as  Services  Provided: 

•  A  mailing  list  of  potential  conference  attendees  was  developed.  (This  is 
planning  for  a  service  that  has  yet  to  result  in  a  conference.  See  definitions 
4.5  and  4.1.3.  This  item  would  be  scored  X.) 

•  Nutrition  education  workshops  will  be  conducted  in  the  month  of  March. 
(This  service  has  not  yet  occurred.  See  definition  4.1.3.  This  entry  would 
be  scored  X.) 


5.    Media  Coverage    —  Abbreviation:  M. 

General  definition:  Coverage  of  the  initiative  or  its  projects  in  the  newspaper, 
radio  or  television,  or  newsletter.  These  may  be  scored  as  (a)  instances  or 
discrete  occurrences  of  coverage,  (b)  column  inches  of  coverage  (for  print 
media),  and/or  (c)  minutes  of  coverage  (for  broadcast  media). 

5.1.  Media  coverage  must  meet  all  of  the  following  criteria: 

5.1.1.  occurred  (not  just  planned) ; 

5.1.2.  be  an  instance  of  radio  time,  television  time,  newspaper 
article,  brochure  or  newsletter;  and 

5.1.3.  feature,  or  be  facilitated  by,  the  initiative. 

5.2.  Record  the  number  of  instances  and  the  extent  of  coverage  (i.e.,  col- 
umn inches  of  print  media  or  minutes  of  broadcast  media)  for  each 
media  exposure.  For  TV  and  radio,  every  airing  of  a  PSA,  news 
report,  or  event  in  which  the  initiative  or  one  of  its  programs  is  men- 
tioned is  counted  as  a  discrete  instance  and/or  in  broadcast  minutes. 
Every  newspaper  article  mentioning  the  initiative  or  program  is 
counted  as  an  instance.   Every  newsletter  article  is  an  instance.  Each 
different  brochure  disseminated  is  an  instance. 

5.3.  Media  coverage  is  counted  if  it  features  the  project,  even  if  the  cover- 
age was  not  initiated  directly  by  the  group.  Airings  and  articles  not 
facilitated  by  the  initiative  are  valid  only  if  the  name  of  the  initiative  or 
one  of  its  projects  or  products  is  mentioned  or  referred  to. 

5.4.  Count  all  instances  of  media  coverage  facilitated  by  the  initiative.  The 
initiative  may  facilitate  media  coverage  in  an  number  of  ways;  for 
example,  by  writing  PSAs,  contacting  editorial  boards,  building  rela- 
tionships with  reporters,  or  sponsoring  media  events. 
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5.5.  Copies  of  print  media  should  be  attached  to  and  stored  with  the  logs. 

5.6.  Internally  produced  media  (such  as  newsletters  and  newsletter  articles) 
are  all  counted  as  media  coverage. 

Examples  of  Media  Coverage: 

•  Newspaper  article  describing  the  initiative  totaling  15  inches  of  space. 
(Scored  as  1  unit  and/or  15  column  inches.   See  definitions  5.1  and  5.2.) 

•  Five  10-minute  radio  spots  describing  the  project  aired  on  the  local  AM 
radio  station.   (Scored  as  5  units  and/or  50  broadcast  minutes.  See 
definitions  5.1  and  5.2.) 

•  Eight  3-minute  radio  spots  describing  the  project  aired  on  the  local  FM 
station.   (Scored  as  8  units  and/or  24  broadcast  minutes.   See  definitions 
5.1  and  5.2.) 

Examples  of  Items  Not  to  be  Considered  as  Media  Coverage: 

•  An  article  on  a  teen  parenthood  prevention  project  in  Washington,  DC, 
public  schools  appeared  in  the  local  newspaper  on  9/12/93.  (This  is  not  an 
instance,  since  the  program  was  not  connected  to  the  initiative.   See 
definitions  5.1.3  and  5.3.  This  entry  would  be  scored  X.) 

•  The  local  health  department  developed  and  distributed  a  free  brochure  on 
preparing  healthy  meals  for  children  on  October  of  1993.   (This  is  not  an 
instance,  since  the  brochure  was  not  facilitated  by  the  initiative.   See 
definition  5.1.3.   Entry  is  scored  X.) 


6.    Resources  Generated  —  Abbreviation:  RC. 

General  definition:  Acquisition  of  funding  for  the  initiative  through  grants, 
donations,  or  gifts  in-kind.  Each  separate  grant  or  donation  is  considered  to  be 
a  unit  of  resources  generated.   Resources  generated  can  include  money, 
materials,  and  people's  time. 

The  monitoring  system  counts  resources  generated  in  two  ways:   units  and 
estimated  value.  Why  count  resources  in  two  ways?  Many  initiatives  will  have  a 
few  large  grants  and  many  small  donations.   Counting  units  (or  instances)  of 
resources  generated  highlights  the  small  donations  that  would  seem 
inconsequential  in  dollar  amount  when  compared  to  the  larger  grants.  These 
small  donations  show  community  support,  however,  and  are  important  to  the 
long-term  financial  sustainability  of  the  initiative. 
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6.1.  Resources  generated  must  meet  all  of  the  following  criteria: 

6.1.1.  occurred  (not  just  planned) ; 

6.1.2.  be  in  the  form  of  money,  materials,  or  donated  professional 
time; 

6.1.3.  be  used  to  facilitate  actions  related  to  the  mission  of  the 
initiative;  and 

6.1.4.  be  allocated  to  the  initiative  (not  one  of  its  partners). 

6.2.  Donation  of  people's  time  is  counted  if  the  person  is  doing  work  they 
are  trained  to  do.   Professional  services  of  builders,  nurses,  teachers, 
and  facilitators  are  examples.   Estimate  the  value  of  the  donated  time 
by  calculating  the  hourly  market  value  of  the  services  multiplied  by  the 
number  of  hours  of  service. 

6.3.  Estimate  the  market  value  of  donated  materials.  For  example,  if  the 
newspaper  donated  advertising  space  for  a  special  event,  determine 
the  market  value  of  that  advertising  space. 

6.4.  Count  grant  moneys  when  they  are  disbursed.  For  example,  if  a  5- 
year  $500,000  grant  was  awarded  and  disbursed  at  $100,000  per 
year,  count  one  instance  of  $100, 000  every  year  over  the  grant 
period. 

Examples  of  Units  of  Resources  Generated: 

•  The  initiative  was  awarded  a  $  1 ,000  grant  from  Kansas  Action  for  Children. 
(New  grant  received.  See  definition  6.1.2.) 

•  Funding  was  received  from  a  foundation  to  implement  nutrition  education 
programs  (Money  generated  with  help  from  the  initiative  for  its  projects. 
See  definition  6.1.) 

•  A  portion  of  a  staff  member's  time  (0.5  full-time  equivalent)  was  assigned  to 
the  partnership  from  the  county  health  department  (Staff  time  was  donated. 
See  definitions  6.1.2  and  6.2.) 

•  A  three-year  implementation  grant  was  awarded  by  the  Kansas  Health 
Foundation.   (New  grant  is  a  resources  generated.  See  definitions  6.1  and 
6.4.) 

•  A  local  physician  volunteered  her  time  to  conduct  Heart  Health  Checks  at  a 
project-sponsored  conference.  (Donation  of  a  person's  professional  time  is 
a  resource  generated.  See  definition  6.2.) 
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Example  of  Items  That  Are  Not  Units  of  Resources  Generated: 

•  The  fund-raising  committee  submitted  a  grant  proposal  to  the  Governor's 
Office.   (This  is  a  planning  product,  since  it  has  not  yet  resulted  in  a  grant. 
See  definitions  6.1.1.  and  3.1.   Entry  would  be  scored  X.) 

•  Project  staff  assisted  with  writing  a  grant  for  the  YWCA  to  secure  funds  to 
build  a  new  track.   (The  money  was  not  allocated  to  the  initiative.   See 
definition  6.1.4.   Obtaining  this  grant  may  be  a  community  change.) 

•  Thirty  volunteers  assisted  with  the  project-sponsored  10-K  run.  (Volunteers 
did  not  donate  professional  time,  according  to  this  entry.  See  definition  6.2. 
This  entry  would  be  scored  X  or  services  provided  if  it  is  the  only  report  of 
the  project  sponsored  run.) 


7.    Not  Scored,  Other  —  Abbreviation:  X. 

General  definition:   Other  items  included  on  the  logs  for  which  no  code  or 
definition  has  been  created.   If  this  occurs,  code  the  item  X. 

New  initiatives  will  often  report  activities  that  are  in  process.   Much  of  the 
monitoring  system,  however,  is  designed  to  track  the  results  of  actions,  such  as 
media  coverage,  services  provided,  and  resources  generated.  Clearly  there  are 
many  activities  that  will  take  place  before  a  result  is  produced;  for  example, 
phone  calls,  planning  meetings,  and  confirmation  memos.  These  activities, 
however,  are  not  categorized  in  this  system. 

7.1.  If  an  item  is  scored  as  an  "X",  it  is  not  also  scored  as  something  else. 

Practice  Exercises  and  Answers:  Assigning  Scoring  Categories 

Included  on  the  next  few  pages  are  practice  logs  and  answer  sheets  for  feedback 
on  practice  scoring.  Following  are  Event  Logs  (Figure  A.l)  and  answer  sheets 
(Figure  A. 2)  and  Services  Provided  Logs  (Figure  A. 3)  and  answer  sheets  (Figure 
A. 4).  Entries  in  the  Media  Coverage  and  Resources  Generated  logs  are  usually 
straightforward,  so  practice  logs  are  not  included.  Practice  is  one  of  the  best 
ways  to  learn  the  monitoring  system.  Entries  listed  further  on  in  the  logs  are 
more  complicated  than  those  listed  in  the  beginning.  Answers  and  explanations 
are  provided  after  each  practice  log  (Figures  A. 2  and  A. 4). 

To  perform  these  exercises,  follow  these  instructions.  Assign  scoring  categories 
to  each  entry  using  the  Expanded  Definitions.   Make  a  photocopy  of  the  logs 
and  write  your  score  in  the  column  labeled  "Code."  Compare  your  scores  with 
the  answers  provided  following  the  sample  logs.  The  practice  logs  are  written  to 
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simulate  logs  you  may  actually  receive,  so  watch  out  for  entries  that  fall  into 
more  than  one  category,  entries  that  are  not  scored  (e.g.,  are  coded  X),  and 
entries  on  the  wrong  forms.  The  monitoring  system  isn't  perfect,  so  practice 
won't  make  you  score  perfectly,  but  it  will  help  increase  your  accuracy! 


Figure  A.1 
Practice  Event  Log 


Event 

Description 

Entry 
No. 

Code 
(CA, 
CC, 
PP, 
SP, 
M, 
RG, 
orX) 

Date 

Describe  the  event  in 
detail.  Include: 

a.  Why  is  it  important? 

b.  What  happened  as  a 
result? 

a.  Who  was  involved? 

b.  What  organizations 
were  collaborators? 

c.  What  community  sector 
or  objective  does  this 
relate  to? 

d.  Was  this  the  first  time 
this  event  happened? 

1. 

1/1/95 

Discussed  enhancing  physical 
activity  levels  and  nutrition 
education  in  the  after-school 
program  at  El  Centra.  Staff 
will  consider  making 
changes. 

a.  El  Centra  staff  and 
Jose  Mendosa,  coalition 
member 

b.  El  Centra 

c.  Broader  community 

d.  No 

2. 

3/1/95 

Met  with  El  Centra  staff. 
Physical  activity  and  nutrition 
components  were  added  to 
the  after-school  program. 

a.  El  Centra  staff  and 
Jose  Mendosa,  coalition 
member 

b.  El  Centra 

c.  Broader  community 

d.  Yes 

3. 

2/6/95 

Met  with  the  manager  of  a 
chicken  place  and  discussed 
adding  low-fat  items  to  their 
menu.  She  agreed  to  contact 
the  franchise  office  to  see 
what  foods  can  be  offered. 

a.  Manager  of  restaurant  and 
staff 

b.  Chamber  of  Commerce 

c.  Business  community 

d.  Yes 

4. 

2/9/95 

Presented  on  the  topic  of 
lower  fat  eating  strategies  at 
the  local  AARP  meeting.   1 6 
people  attended.  They  asked 
great  questions  about  cooking 
techniques. 

a.  Coalition  member  Kristie 
arranged,  Chef  Jones 
presented 

b.  AARP 

c.  Broader  community 

d.  Yes 
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Figure  A.1 
Practice  Event  Log  (Continued) 


Event 

Description 

Code 

a.  Who  was  involved? 

(CA, 

b.  What  organizations 

CC, 

were  collaborators? 

PP, 

Describe  the  event  in 

c.  What  community  sector 

SP, 

detail.  Include: 

or  objective  does  this 

Entry 

M, 

a.  Why  is  it  important? 

relate  to? 

No. 

RG, 

b.  What  happened  as  a 

d.  Was  this  the  first  time 

orX) 

Date 

result? 

this  event  happened? 

5. 

3/3/95 

Wrote  an  editorial  on  our 
need  for  bike  paths  along  the 
river.  The  editorial  will  be 
featured  in  Sunday's  Herald 
Times. 

a.  Media  Task  Force  member 
Ella  wrote  it 

b.  YWCA  helped  with  the 
research 

c.  Broader  community 

d.  First  time  Ella  wrote  one 

6. 

2/1/95 

Tonya,  Associate  Professor  at 
Haskell  Indian  Nations  Uni- 
versity, donated  2  hours  of 
her  time  analyzing  the 
BRFSS's  data  for  the  Pinkney 
neighborhood. 

a.  Tonya 

b.  Haskell 

c.  Not  sure 

d.  No 

7. 

1/8/95 

Met  with  parents  from  the 
Waldorf  school  to  finalize 
school  snack  and  lunch 
menus.    Parents  rewrote  the 
menu  to  include  lower  fat  and 
higher  fiber  foods.  New 
menus  are  being  used  and  the 
kids  are  eating  most  of  the 
food! 

a.  Parents  and  staff 

b.  Nutritionist  on  Health 
Task  Force  helped 

c.  School  sector 

d.  Yes,  for  the  Waldorf 
school 

8. 

2/4/95 

Met  with  St.  John  elementary 
school  and  School  Food  Ser- 
vice director  and  discussed 
changing  the  school  food  pro- 
gram. He  is  going  to  talk  to 
the  principal  and  we  will  meet 
again  next  week. 

a.  Staff 

b.  St.  John  school 

c.  School  sector 

d.  Yes 
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Figure  A.1 
Practice  Event  Log  (Continued) 


Event 

Description 

Entry 
No. 

Code 
(CA, 
CC, 
PP, 
SP, 
M, 
RG, 
orX) 

Date 

Describe  the  event  in 
detail.  Include: 

a.  Why  is  it  important? 

b.  What  happened  as  a 
result? 

a.  Who  was  involved? 

b.  What  organizations 
were  collaborators? 

c.  What  community  sector 
or  objective  does  this 
relate  to? 

d.  Was  this  the  first  time 
this  event  happened? 

9. 

2/5/95 

Arranged  a  meeting  with  the 
Chamber  of  Commerce  and 
MAS  (Mothers  Against 
Smoking).  This  is  the  first 
time  these  groups  have  met. 
MAS  asked  business  leaders 
to  adopt  at  least  one  practice 
change  that  would  make  ciga- 
rettes more  difficult  for  youth 
to  obtain.  They  were  really 
interested.  We  scheduled 
another  meeting  to  formalize 
agreements. 

a.  MAS,  the  Chamber,  and 
staff 

b.  See  above 

c.  Business  community  and 
broader  community 

d.  Yes 

10. 

2/6/95 

The  project  held  a  commu- 
nity-wide listening  session  to 
hear  people's  concerns  about 
secondhand  smoke.  Commu- 
nity member's  ideas  were 
added  to  an  action  plan  to 
address  the  problem. 

a.  36  community  members 
and  all  staff 

b.  Community  building 
donated  space 

c.  Broader  community 

d.  First  on  this  topic 

11. 

2/7/95 

The  Health  Organizations 
Task  Force  met.  Discussed 
priorities  and  leadership 
needs.  Scheduled  meetings 
for  next  6  months. 

a.  Task  Force  members 

b.  None 

c.  Health  Organizations 

d.  No 
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Figure  A.1 
Practice  Event  Log  (Continued) 


Event 

Description 

Code 

a.  Who  was  involved? 

(CA, 

b.  What  organizations 

CC, 

were  collaborators? 

PP, 

Describe  the  event  in 

c.  What  community  sector 

SP, 

detail.  Include: 

or  objective  does  this 

Entry 

M, 

a.  Why  is  it  important? 

relate  to? 

No. 

RG, 

b.  What  happened  as  a 

d.  Was  this  the  first  time 

orX) 

Date 

result? 

this  event  happened? 

12. 

2/9/95 

Worksite  Task  Force  meeting 
was  held.  Joe,  who  is  in 
change  of  PR  at  the  hospital, 
agreed  to  include  the  project's 
brochure,  "Heart  Health  Tips" 
on  all  meal  trays.  The  bro- 
chures are  now  put  on  trays 
during  the  first  week  of  every 
month. 

a.  Task  Force  members 

b.  None 

c.  Health  Organizations 

d.  No,  they  meet  monthly 

13. 

2/9/95 

Superintendent  of  Haskell 
Indian  Nations  University 
told  me  at  church  that  they 
changed  the  school  policy  to 
require  physical  education, 
stressing  lifelong  exercise, 
for  all  students. 

a.  Haskell  Indian  Nations 
University 

b.  None 

c.  Schools 

d.  Yes 
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Figure  A.2 
Answer  sheet  for  practice  Event  Log 


Entry 
No. 

Correct 
Code(s) 

Explanation 

Definition 
No. 

1. 

CA 

The  purpose  of  this  meeting  is  to  change  the  after- 
school  program,  which  is  a  change  in  El  Centra's 
practices.    The  score  of  community  change,  however,  is 
not  appropriate  because  the  change  has  not  happened 
yet.  The  meeting  is  a  precursor  to  the  desired 
community  change. 

1.1 

2. 

CAand 
CC 

The  change  discussed  in  #1  happened  and  is  scored  CC. 
The  meeting  reported  in  this  entry  is  scored  as  CA 
because  it  meets  all  the  criteria  listed  in  1.1.   Most 
community  changes  are  accompanied  by  an  additional 
score,  usually  CA  or  SP 

2.1  and 
1.1 

3. 

CA 

The  purpose  of  this  meeting  is  to  change  the  menu, 
which  would  be  a  change  in  a  Chicken  Place  practices. 
As  with  #1,  the  change  has  not  happened  yet.  Notice 
that  the  recorded  answered  "yes"  to  the  question  "Was 
this  the  first  time  this  event  happened?"  Answers  to  this 
question  can  be  misleading.  This  may  be  the  first 
meeting  with  the  restaurant,  but  not  the  first  instance  of 
a  change  in  program,  policy,  or  practice. 

1.1  and 
2.1.1 

4. 

SP 

The  purpose  of  this  event  is  to  inform  people  about 
eating  less  fat,  which  falls  under  the  category  of  services 
provided.  Notice  that  this  service  was  put  in  the  event 
log.  The  event  log  is  designed  to  record  community 
actions  and  community  changes,  but  it  may  be  used  to 
report  events  that  fall  in  other  categories. 

4.1.1 

5. 

X 

The  purpose  of  writing  the  editorial  is  to  produce  media 
coverage.  After  it  has  been  published,  the  editorial  will 
be  an  instance  of  media  coverage. 

5.1.1 

6. 

RG 

1  unit 
$70 

Tonya  donated  her  professional  time,  which  is  an 
instance  of  resource  generation.  The  value  of  her  time 
was  estimated  at  $35.00  per  hour.  A  telephone  call  to 
determine  the  going  rate  for  a  service  or  an  educated 
guess  can  be  used  to  estimate  value  of  someone's  time. 

6.2 
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Figure  A. 2 
Answer  sheet  for  practice  Event  Log  (Continued) 


Entry 
No. 

Correct 
Code(s) 

Explanation 

Definition 
No. 

7. 

CAand 
CC 

The  Waldorf  school  changed  the  practice  involved  in 
preparing  food  for  the  children  at  school,  which  is  a 
community  change.  The  meeting  to  make  this  change  is 
scored  as  a  CA. 

1.1  and 
2.1 

8. 

CA 

The  purpose  of  the  meeting  is  to  change  the  food 
service  practices  at  St.  John  school.  The  change  has  not 
happened  yet. 

1.1 

9. 

CC  and 
CA 

CA 

This  is  another  example  of  a  number  of  events  in  one 
entry.  Two  new  groups  collaborated  to  make  changes  in 
the  community.  The  new  collaboration  is  a  change  in 
practice  and  is  scored  as  a  community  change.  The 
meeting  itself  is  a  CA. 

MAS  and  the  initiative  requested  that  businesses  adopt 
new  practices  making  cigarettes  more  difficult  for  youth 
to  obtain.  The  purpose  of  the  request  is  a  practice 
change  that  has  not  yet  happened. 

2.4  and 
1.1 

1.1 

10. 

CA 

RG 

1  unit 
$30.00 

The  purpose  of  this  presentation  is  to  gather 
information  from  community  members  to  make  changes 
related  to  reducing  secondhand  smoke.  Ideas  were 
added  to  the  group's  action  plan. 

The  community  building  space  was  donated.  The 
donation  is  an  instance  of  a  resource  generated. 
Donation  was  estimated  to  be  valued  at  $30. 

1.2 
6.3 

11. 

X 

This  was  an  internal  meeting.  The  meeting  did  not 
result  in  any  planning  products. 

1.4  and 
3.8 

12. 

CAand 
CC 

This  was  an  internal  meeting.  It  is  different  from  #11  in 
that  a  Task  Force  member  was  acting  in  the  role  of  PR 
coordinator  for  the  hospital.   A  community  change 
occurred  during  the  internal  meeting. 

1.4.1  and 
2.1 

13. 

X 

As  reported  in  this  entry,  the  coalition  did  not  facilitate 
this  community  change. 

2.1.5 
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Figure  A.3 
Practice  Ongoing  Services  Provided  Log 


Entry 

No. 

Code 

Date 

Service 
(e.g.,  workshop, 
class,  screening) 

Location 

No.  of 

People 

Attending 

No. 

of 

Hours 

New 
Service? 
(Yes/No) 

1. 

1/1/95 

Cholesterol  screenings 

Northridge 
nursing  home 

76 

5 

No 

2. 

3/1/95 

Cool  Guys  Thursday  Night 
walk-in  group 

Hard  Rock 
Cafe 

16 

1 

No 

3. 

2/6/95 

Cook-off  featuring 
low-fat  ethnic  foods 

Southwest 
AME  church 

134 

3 

No 

4. 

2/9/95 

Presented  on  the 
mission  of  the  project 
and  evaluation  data 
to  the  United  Way 
board  of  directors 

United  Way 

5 

1 

Yes 

5. 

3/3/95 

Presentation  to  MADD 
on  steps  they  can  take 
to  increase  enforcement 
of  Synar  ordinances 
(no  sales  to  minors).  Will 
meet  next  week  to  develop 
a  collaborative  action  plan. 

Home  of  Mary 
Mathews 

15 

1 

Yes 

6. 

2/1/95 

Presented  tips  on  how 
to  increase  time  spent 
in  lifelong  aerobic  physical 
activities  in  high  school 
PE  classes  to  a  group  of 
PE  teachers.  Principal  is 
interested  in  changing  PE 
class  curriculum  and  wants 
PE  teacher's  support. 

Jackson  High 
School 

35 

2 

No 

7. 

1/8/95 

Smoking  cessation  sup- 
port groups  for  pregnant 
teens  held  on  Tuesday 
and  Thursday 

Johnson 
High  School 
and  Bryant 
High  School 

Tu-5 
Th-7 

1 

No 

8. 

2/4/95 

First  Annual  breast  cancer 
walk-a-thon 

Downtown 

360 

4 

Yes 

9. 

2/6/95 

Met  with  SADD  chapter 
members  to  plan  next 
STOP  class 

Office 

3 

1 

No 
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Figure  A.3 
Practice  Ongoing  Services  Provided  Log  (Continued) 


Entry 
No. 

Code 

Date 

Service 
(e.g.,  workshop, 
class,  screening) 

Location 

No.  of 

People 

Attending 

No. 

of 

Hours 

New 
Service? 
(Yes/No) 

10. 

2/8/95 

Lawrence  Memorial 
Hospital  sponsored 
6  smoking  cessation 
classes 

LMH 

7 

? 

Yes 

11. 

3/7/95 

Presented  the  injury 
prevention  advantages 
of  designated  bike  paths 
to  a  local  physician's  group. 
They  decided  to  cosponsor 
a  local  ordinance  supporting 
bike  paths. 

LMH 

6 

2 

Yes 
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Figure  A.4 
Answer  sheet  for  practice  Ongoing  Services  Provided  Log 


Entry 

No. 

Correct 
Code(s) 

Explanation 

Definition 
No. 

1. 

SP 

The  purpose  of  the  screenings  was  to  provide  information 
about  cholesterol  levels  and  general  education  on  risk 
reduction. 

4.1 

2. 

SP 

The  purpose  of  the  group  is  to  provide  support  for 
participating  in  physical  fitness  activities. 

4.1.1 

3. 

SP 

The  purpose  of  the  cook-off  is  to  enhance  people's  skills  in 
low-fat  cooking. 

4.1 

4. 

SP 

At  first  glance  the  event  looks  like  it  might  be  a  community 
action.  The  presentation,  however,  was  not  interactive 
nor  aimed  specifically  at  some  community  change; 
therefore  it  is  not  a  community  action. 

1.2  and 
4.1 

5. 

CA 

The  presentation  was  aimed  at  a  specific  community 
change,  enforcement  of  an  ordinance;  therefore  the  event 
is  a  community  action. 

1.2 

6. 

CA 

This  event  is  similar  to  #5.  The  purpose  is  to  change  the 
practice  of  PE  teachers. 

1.2 

7. 

SPand 
SP 

Support  groups  are  services.   Note  that  two  services  are 
reported  in  one  entry. 

4.4 

8. 

CC  and 
SP 

The  walk-a-thon  is  a  new  practice  in  the  community  and  is 
therefore  scored  as  a  community  change  and  service 
provided. 

2.1  and 
4.2 

9. 

X 

Planning  for  services  is  not  a  service  provided  even  if 
people  external  from  the  group  participate  in  class 
preparation. 

4.5 

10. 

X 

This  entry  suggests  that  the  initiative  did  not  sponsor  or 
facilitate  the  classes. 

4.1.2 

11. 

CA 

The  purpose  of  the  presentation  was  to  secure  support  for 
a  specific  community  change;  it  is  therefore  a  community 
action. 

1.2  and 
4.3 

CVD  Evaluation  Handbook 


155 


Assessing  Reliability  of  Scoring 


This  section  addresses: 

•  Steps  for  assessing  reliability. 

•  What  to  do  with  low  scores. 

•  Examples. 

•  Practice  exercises  and  answers. 


Steps  for  Assessing  Reliability 

Some  partnerships  may  intend  to  publish  or  make  conference  presentations  of 
monitoring  data.   In  these  circumstances,  it  may  be  important  to  assess 
reliability  on  the  scoring  of  coded  logs.  Reliability  refers  to  whether  independent 
observers  score  events  in  the  same  way.  For  those  initiatives  with  few  resources 
or  less  stringent  requirements  for  evaluation,  assessing  reliability  may  be 
unnecessary. 

There  are  only  three  steps  taken  to  calculate  reliability:   scoring,  rilling  out  a 
reliability  table,  and  calculating  the  agreement  score. 

Step  1 :  Two  People  Score  the  Same  Set  of  Logs.  Two  people  independently 
categorize  the  same  log  forms.  One  person  acts  as  a  primary  observer  and  one 
is  a  reliability  observer.  A  reliability  observer  is  someone  who  independently 
scores  logged  events  to  ensure  that  events  are  coded  accurately  in  accordance 
with  the  written,  agreed-upon  definitions.   If  new  definitions  or  categories  are 
developed  or  revised  based  on  discussions  between  the  primary  and  reliability 
observers,  recode  the  logs  separately  and  recalculate  reliability  (you  may  have  to 
go  back  to  the  beginning  of  the  logs).  A  random  sample  of  items  may  be  used 
for  reliability  for  those  months  when  many  logs  are  returned  (10%  to  20% 
would  be  reasonable,  although  evaluators  might  do  100%  for  the  first  few 
months  of  the  evaluation). 

Step  2:  Fill  Out  a  Reliability  Table.  The  primary  observer  obtains  the  scored 
logs  from  the  reliability  observer  and  compares  them  with  his  or  her  own  by 
using  a  reliability  table.   The  reliability  table  can  be  as  simple  as  a  table  listing 
each  event  and  the  codes  of  both  observers.  The  codes  are  compared,  and 
agreements  and  disagreements  are  noted. 
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Step  3:  Calculate  the  Reliability  Score.  Reliability  scores  range  from  perfect 
agreement  (100%)  to  complete  disagreement  (0%).   Researchers  disagree  on 
the  range  of  acceptable  levels  of  reliability,  but  scores  over  80%  are  generally 
accepted.  There  are  a  number  of  formulas  that  can  be  used  to  calculate 
reliability.   The  following  formula  assesses  reliability  generously.  A  more 
conservative  formula  may  be  used  if  the  data  are  shared  with  scientific 
audiences. 

A  reliability  score  can  be  calculated  by  totaling  each  cell  in  the  reliability  table 
and  using  the  formula: 

reliability  score  =  (no.  of  agreements/total  no.  of  coded  events)  X  100 

What  To  Do  If  Your  Reliability  Scores  Are  Low 

Lower  rates  of  reliability  are  expected  at  first.   Don't  worry:  with  practice 
reliability  scores  will  probably  increase.  If  not,  discuss  events  difficult  to  score 
with  the  reliability  observer.  Try  to  reach  consensus  on  the  appropriate  category 
for  these  events.  Add  these  events  difficult  to  score  under  examples  in  the 
monitoring  definitions.  If,  after  practicing  and  talking,  reliability  scores  are  still 
low,  consider  assigning  someone  with  the  role  of  "expert."  The  expert  should 
have  the  best  grasp  of  the  definitions.   Everyone  else  can  then  try  to  calibrate 
their  scores  to  the  categories  assigned  by  the  expert.  If  all  else  fails,  modify  the 
definitions  or  add  new  categories. 

Example:  Calculating  Reliability 

Helen  scores  this  initiative's  logs  regularly;  she  is  the  primary  observer.  Vince, 
who  has  been  scoring  logs  from  another  initiative,  codes  the  same  set  of  logs  to 
assess  the  accuracy  of  their  coding.  Vince  is  the  reliability  observer.  The  scores 
from  both  of  them  are  listed  in  Figure  A. 5. 
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Figure  A.5 
Codes  for  example  set  of  logs 


Helen's 

Vince's 

Agreement  or 

Event 

Code 

Code 

Disagreement 

1 

CA 

CA 

A 

2 

CA 

PP 

D 

3 

CA 

CA 

A 

4 

CA 

X 

D 

5 

SP 

CA 

D 

6 

CC 

CC 

A 

7 

SP 

SP 

A 

8 

SP 

SP 

A 

9 

SP 

SP 

A 

10 

SP 

SP 

A 

11 

SP 

SP 

A 

12 

X 

X 

A 

13 

M 

M 

A 

14 

CC 

CC 

A 

15 

M 

M 

A 

16 

X 

X 

A 

17 

CA 

X 

D 

18 

CA 

SP 

D 

19 

CA 

CA 

A 

20 

CA 

CA 

A 

21 

X 

X 

A 

22 

X 

X 

A 

23 

CC 

CC 

A 

24 

X 

X 

A 

25 

X 

X 

A 

26 

RG 

RG 

A 

27 

RG 

RG 

A 

28 

X 

X 

A 

29 

M 

M 

A 

30 

M 

M 

A 

The  column  on  the  far  right  in  Figure  A.5  indicates  Helen  and  Vince's 
agreement  or  disagreement  on  which  category  the  event  should  be  assigned: 
"A"  indicating  agreement  and  "D"  for  disagreement.  To  calculate  reliability,  the 
total  number  of  agreements  is  divided  by  the  number  of  agreements  plus 
disagreements  and  the  result  is  multiplied  by  100. 
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Helen  and  Vince  scored  25  of  the  events  the  same.  They  scored  a  total  of  30 
events. 

The  reliability  of  their  scores  is  calculated  as  follows: 

reliability  score  =  (25/30)  X  100. 

Helen  and  Vince  obtained  a  reliability  score  of  83.3%,  an  acceptable  level  of 
reliability. 

Practice  Exercises  and  Answers:  Assessing  Reliability 

The  exercises  in  this  section  will  give  you  practice  in  taking  two  people's  scores 
for  a  set  of  logs  and  calculating  the  reliability  of  their  scoring.  To  perform  these 
exercises,  follow  these  instructions.  Indicate  if  the  primary  and  reliability 
observer  agreed  or  disagreed  on  the  score  they  assigned  to  each  event  in  Figure 
A. 6.  Use  the  formula  provided  in  this  chapter  ([no.  of  agreements/total  no.  of 
events]  X  100)  to  calculate  the  reliability  score. 
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Figure  A.6 
Codes  for  practice  exercises 


Codes  of 

Codes  of 

Event 

Primary 

Reliability 

Agreement  or 

Observer 

Observer 

Disagreement 

1 

SP 

CA 

2 

CA 

PP 

3 

CA 

CA 

4 

CA 

X 

5 

SP 

CA 

6 

CC 

CC 

7 

X 

CC 

8 

CC 

CC 

9 

SP 

SP 

10 

CA 

CA 

11 

SP 

SP 

12 

X 

X 

13 

M 

M 

14 

CC 

CC 

15 

M 

M 

16 

X 

X 

17 

SP 

SP 

18 

X 

X 

19 

M 

M 

20 

CA 

CA 

21 

X 

X 

22 

X 

X 

23 

CC 

CC 

24 

X 

X 

25 

X 

X 

26 

RG 

RG 

27 

RG 

RG 

28 

X 

X 

29 

CC 

CC 

30 

M 

M 

Complete  the  following  information  and  calculate  the  reliability  score  below. 

•  Number  of  agreements: 

•  Total  number  of  events  (agreements  +  disagreements): 

•  Reliability  score  (percent  agreement):  
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Now  compare  your  responses  with  the  correct  answers  below. 

•  Number  of  agreements:   25 

•  Total  number  of  events:    30 

•  Reliability  score  (percent  agreement) :     83. 3% 


Graphing  and  Providing  Feedback 


This  section  provides: 

Steps  for  graphing  and  providing  feedback. 

An  example. 

Practice  exercises  and  answers. 


Steps  for  Graphing  and  Providing  Feedback 

After  a  set  of  logs  is  completely  coded,  the  evaluators  graph  the  data.   In 
assessing  reliability,  it  is  important  to  graph  only  the  primary  observer's  scoring 
categories.  There  are  many  ways  of  graphing,  but  the  best  way  is  to  display  it  so 
that  people  understand  it.   This  will  take  some  trial  and  error  with  each 
particular  initiative.   (Line  graphs,  bar  graphs,  colored  lines,  etc.,  are  all 
possible).    Graphs  can  be  constructed  by  hand  or  with  a  computer. 

We  recommend  preparing  cumulative  line  graphs,  as  shown  in  Figure  4.5.  By 
adding  new  events  to  previous  ones,  cumulative  graphs  provide  a  picture  of 
ongoing  development  of  the  initiative.  The  slope  of  the  line  is  an  indicator  of 
how  much  activity  or  accomplishment  occurs.  The  steeper  the  slope,  the  more 
activity;  a  flat  line  shows  no  activity. 

Graphing  is  quite  easy,  but  it  can  be  time  consuming.   Constructing  graphs  by 
hand  is  tedious,  and  learning  a  new  computer  program  requires  an  initial  time 
investment.   Graphs  are  worth  the  time!  After  recording  and  categorizing  all 
these  events,  why  stop  short  of  the  most  informative  and  useful  product? 
Graphing  makes  the  data  collected  easier  to  understand  and  provides  a  way  to 
organize  years  of  data. 

Graphing  and  providing  feedback  is  a  three-step  process.  First,  tally  scores  by 
category  and  add  the  number  to  the  previous  months'  data.  Second,  graph  the 
total  number  of  events  in  each  category.  Finally,  share  and  discuss  the  graphs 
with  members  of  the  initiative  and  community  members. 
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Step  1 :  Tally  Scores  and  Add  to  Previous  Data.  Add  up  the  number  of  events  in 
each  category  (e.g.,  number  of  community  actions  and  number  of  community 
changes)  scored  in  the  current  set  of  logs.  Note  that  it  is  important  that  only  the 
data  from  the  primary  observer  are  graphed.   Otherwise,  the  data  will  be 
distorted  and  the  reliability  scores  will  be  meaningless.  Enter  the  numbers  on  a 
tally  sheet  that  contains  similar  data  from  previous  sets  of  logs. 

Step  2:  Graph  the  Data.  Locate  the  point  on  the  graph  that  represents  the 
cumulative  number  of  each  category,  draw  a  dot,  and  connect  the  new  dot  to 
the  dot  that  represents  the  previous  total.    That's  all  there  is  to  it. 

Step  3:  Feed  Data  Back  to  Members  of  the  Initiative.  Once  the  graphs  are 
completed,  make  a  list  of  the  community  changes,  and  lists  of  items  for  any  of 
the  other  codes,  for  the  current  logs.  These  lists  are  extremely  helpful  for  those 
explaining  the  graphs  and  saves  them  the  trouble  of  reading  logs  that  may  not 
contain  many  actions  or  outcomes.  The  lists  should  be  updated  monthly,  and 
outcomes  should  be  grouped  by  month. 

Share  the  graphs  with  staff  from  the  initiative  and  community  members. 


Presentation  of  monitoring  graphs 
should  include  the  following 

Introducing  the  coding  categories:  provide  examples  of  each  category  and 
state  why  this  measure  is  important. 

Pointing  out  what  is  positive  about  the  data. 

Noting  trends  in  the  line  and  explaining  the  possible  meaning  of  the  trends. 


Example:  Graphing  and  Providing  Feedback 

Data  from  Helen's  scores  (taken  from  Figure  A. 5)  are  used  to  illustrate  how  to 
graph  data  cumulatively. 

Figure  A. 7  shows  an  example  tally  sheet.  Data  for  each  month  are  listed  on  the 
row  marked  "cum."  for  cumulative.   Monthly  data  are  added  to  the  previous 
month's  data  in  this  row.  For  example,  there  were  8  community  actions  in 
August.   Eight  was  added  to  the  cumulative  total  of  30  in  July  for  a  new 
cumulative  total  of  38  in  August. 
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Figure  A. 8  shows  a  cumulative  graph  of  the  community  actions  tallied  in  Figure 
A. 7.  Notice  the  flat  line  between  January  and  March.  In  cumulative  graphs,  flat 
lines  indicate  no  activity:  no  community  actions  were  produced  in  February  and 
March.  The  line  becomes  steeper  starting  in  April  and  continuing  through 
August,  which  indicates  more  community  actions  were  produced  during  that 
period. 

Figure  A.8 
Example  graph  of  community  actions  plotted  cumulatively 
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Practice  Exercises  and  Answers 

The  exercise  that  follows  provides  practice  in  summarizing  and  graphing  four 
months  of  monitoring  data.  Figure  A. 9  lists  sample  scoring  categories  for 
September-December.  Use  the  Tally  Sheet  provided  in  Figure  A.  10  to  calculate 
cumulative  totals  for  these  months.   Graph  the  results,  for  community  changes 
only,  on  Figure  A.  11.   Finally,  provide  answers  to  questions  about  the  graph. 

Figure  A.9 
Sample  data  for  practice  exercise 


Sept. 

PP 

CC 

cc 

CA 

CA 

CA 

SP 

SP 

CC 

M 

M 

CC 

CA 

Oct. 

CA 

SP 

SP 

SP 

CA 

CA 

CC 

CC 

CC 

CC 

M 

CA 

CA 

Nov. 

CC 

cc 

SP 

SP 

SP 

CA 

CA 

CA 

CA 

CA 

M 

M 

M 

Dec. 

cc 

SP 

SP 

SP 

CA 

CA 

PP 

SP 

SP 

SP 

SP 

CC 

CC 

164 


CVD  Evaluation  Handbook 


O  .w 

<  m 
o>  a. 

_>> 

/5 


3- 

u 


6 
Q 

> 

o 

2 

o 

o 

CO 

3 
< 

00 

00 
00 

co 

i— i 
i—i 

^0 

CO 

o 

CO 
i—l 

<* 

LO 
CM 

O 

LO 

oo 

o 
o 

i—i 

i—i 
o 

1—1 

CM 

CO 

1—1 

3 

"-3 

1—1 

o 

CO 

^h 

00 

00 

o 

CO 

CM 

CO 

1—1 

o 

O 

o 

CO 

o 

LO 

o" 
o 

1—1 

1—1 

1—1 
1— 1 

0) 
C 

a-N 

On 

CM 

<vf 

l> 

CM 

i—i 

1— 1 
1—1 

1—1 

i—i 
CM 

o 
o 

o 

CM 

o~ 

o 

1—1 

\D 

o 

1—1 

> 

to 
2 

<o 

O 
CM 

i—i 

CM 

[> 

O 
CM 

o 

o 

1— I 

lO 

O 
CM 

O 
CM 

o 

CM 

i—i 

o" 
o 

1— 1 

1—1 

^ 

a 
< 

-* 

i—i 

i—i 

i—i 

VO 

CO 
i— 1 

o 

o 

1—1 

CO 

LO 
i—l 

o 
o 

1— 1 

o 
o 

1—1 

o 

1—1 

CM 

CO 

V-' 

(0 
2 

o 

O 
i— i 

o 

O 

^ 

r-- 

CM 

o 

1—1 

CM 

CM 

i—i 

o 

o 
o 
o 
o~ 

o 

T— 1 

o 

1—1 

o 

o 

1— t 

o 

o 

CO 

oo 

<vt< 

00 

LO 

O 
i—l 

o 

o 
o 
o 

o" 

o 

1—1 

o 

1—1 

c 

(0 

*-3 

o 

1—1 

o 

1—1 

o 

o 

o 

o 

st 

«tf 

LO 

LO 

o 
o 
o 

o" 

o 

1—1 

o 
o 
o 

o~ 

o 

1—1 

1— 1 

1— 1 

6 
2 

E 

3 

u 

d 
2 

e 

3 

u 

6 
2 

e 

3 

u 

6 
2 

3 

u 

6 
2 

E 

3 

u 

6 
2 

£ 

3 

u 

6 
2 

£ 

3 

u 

< 
U 

U 
U 

CO 

to 

c 

3 

o 

> 

^3 
_2 

3 

s 

3 

u 


3 

u 


CVD  Evaluation  Handbook 


165 


Figure  A.11 
Graph  for  practice  exercise 
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Looking  at  the  graph  you  constructed  on  community  changes,  answer  the 
following  questions. 

Are  there  any  flat  points  on  the  line  representing  community  changes?  If  so, 
during  which  months?  

Are  there  any  steep  slopes  on  the  line  representing  community  changes?  If  so, 
during  which  months?  

What  conclusions  can  you  draw  about  community  changes  facilitated  over  this 
period? 
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Figure  A.1 2 
Answer  Tally  Sheet  for  practice  exercise 
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Figure  A.13 
Answer  graph  for  practice  exercise 
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Are  there  any  flat  points  on  the  line  representing  community  changes?  If  so, 
during  which  months?  There  is  a  flat  line  from  January  through  March. 

Are  there  any  steep  slopes  on  the  line  representing  community  changes?  If  so, 
during  which  months?  Steepest  part  of  the  line  is  between  July  and  October. 

What  conclusions  can  you  draw  about  community  changes  produced  over  this 
period?  No  community  changes  were  produced  at  the  beginning  of  the  year. 
Perhaps  this  is  a  new  initiative.    The  first  community  change  happened  in 
April,  and  there  is  a  steady  increase  of  community  changes  produced  until 
November.    There  was  a  slight  drop  in  the  slope  in  November.   Perhaps  the 
holiday  season  resulted  in  a  slower  rate  of  change. 


168 


CVD  Evaluation  Handbook 


Appendix  B 


Blank  Forms  for  the 
Monitoring  System 


This  appendix  provides  blank  forms  for  implementing  the  monitoring  system. 


This  appendix  includes: 

•  Event  Log. 

•  Ongoing  Services  Provided  Log. 

•  Media  Coverage  Log. 

•  Resources  Generated  Log. 

•  Tally  Sheet  for  summarizing  monitoring  data. 
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Appendix  C 


Portions  of  the 

1 995  Behavioral  Risk  Factor 

Questionnaire  that  relate  to  CVD 
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1995  Behavioral  Risk  Factor  Questionnaire 


FIPS  STATE        STRATUM 
CODE                 CODE 

PSU 
NUMBER 

RECORD 
NUMBER 

DATE  OF  INTERVIEW 

MM                 DD                YY 

ID 

(1-2) 

(3) 

(4-8) 

(9) 

(10-15) 

(16-17) 

o 


HELLO,  I'm calling  for  the_ 

We're  doing  a  study  of  the  health  practices  of. 


residents. 


to  be 


Your  phone  number  has  been  chosen  randomly  by  the 

included  in  the  study,  and  we'd  like  to  ask  some  questions  about  things  people  do  which  may  affect 
their  health. 


O 


Is  this 


(18-20) 


Area  code 

Prefix 

Suffix 

(21-23) 


(24-25) 


Thank  you  very  much,  but  I  seem  to 
have  dialed  the  wrong  number.  It's 
possible  that  your  number  may  be 
called  at  a  later  time.    STOP 


o 


Is  this  a  private  residence? 


Thank  you  very  much,  but  we  are  only 
interviewing  private  residences.    STOP 


Date 


0 
Line  busy 


No  answer 


o 
o 
o 
o 
o 


Time 


o 
o 
o 
o 
o 


Time 


o 
o 
o 
o 
o 


Time 


ID 


Comments 


Appointments: 

Today's  date/time 

1.      

2.       


Spoke  with 


Ask  for 


Callback 
date/time 


ID 


Comments 


Refusals: 


1st 


2nd 


Date/time 


Spoke  with 


ID 


Comments 


Call  Disposition  Codes 


01  -  Completed  interview. 

02  -  Refused  interview. 

03  -  Nonworking  number. 

04  -  No  answer  (multiple  times). 

05  -  Business  phone. 

06  -  No  eligible  respondent 

at  this  number. 


07  -  No  eligible  respondent  could  be  reached  during 

time  period. 

08  -  Language  barrier  prevented  completion  of 

interview. 

09  -  Interview  terminated  within  questionnaire. 

10  -  Line  busy  (multiple  tries). 

1 1  -  Selected  respondent  unable  to  respond 

because  of  physical  or  mental  impairment. 
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Edited  by: 
Date: 


(28) 


Final  disposition 
of  telephone  call: 

(26-27) 

Wind  down: 

183 


o 


o 


Our  study  requires  that  we  randomly  select  one  adult  who  lives  in  your  household 
to  be  interviewed.  How  many  members  of  your  household,  including  yourself,  are 
1 8  years  of  age  or  older? 


(29-30) 


If  "1"£^  Are  you  the  adult? 

If  "yes"£^  Then  you  are  the  person  I  need  to  speak  with.  Go  to  page  3  P^3 

If  "no"  £^  May  I  speak  with  him  or  her?  Go  to  "correct  respondent"  at  bottom  of  page 

Men  Women 

How  many  of  these  adults  are  men  and  how  many  are  women? 


(31) 


Who  is  the  oldest  man  who  presently  lives  in  this  household? 
Who  is  the  next  oldest  man  who  presently  lives  in  this  household? 


Etc. 


Who  is  the  oldest  woman  who  presently  lives  in  this  household? 
Who  is  the  next  oldest  woman  who  presently  lives  in  this  household? 


Etc. 


(32) 


Suffix: 


1. 

2. 

3. 

4. 
Total 
adults     5. 

6. 
7. 
8. 


Name  or  Relationship 


Last  digit  of  phone  number 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

3 

1 

2 

X 

1 

2 

3 

4 

1 

2 

3 

4 

X 

X 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

5 

6 

1 

2 

3 

4 

X 

X 

X 

X 

2 

3 

4 

5 

6 

7 

1 

X 

X 

X 

8 

1 

2 

3 

4 

5 

6 

7 

X 

X 

1. 

2. 

3. 

4. 

Total 
5.      adults 

6. 
7. 
8. 


o 


The  person  in  your  household  that  I  need  to  speak  with  is_ 


To  correct  respondent 


o 


If  "you,"  go  to  page  3  C^T* 


Hello,  I'm, 
the 


calling  for 

I'm  a  member  of  a 


special  research  team.  We're  doing  a  study  of_ 


residents  regarding  their  health  practices  and  day-to-day  living 
habits.  You  have  been  randomly  chosen  to  be  included  in  the  study 
from  among  the  adult  members  of  your  household. 
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Section  3:  Hypertension  Awareness 

9.  About  how  long  has  it  been  since  you  last  had  your  blood  pressure 

taken  by  a  doctor,  nurse,  or  other  health  professional?  (44) 

Read  Only  if  Necessary 

a.  Within  the  past  6  months  (1  to  6  months  ago) 1 

b.  Within  the  past  year  (6  to  12  months  ago) 2 

c.  Within  the  past  2  years  (1  to  2  years  ago) 3 

d.  Within  the  past  5  years  (2  to  5  years  ago) 4 

e.  5  or  more  years  ago  5 

Don't  know/Not  sure 7 

Never    Go  to  Q.  12  (p.  8) 8 

Refused 9 

10.  Have  you  ever  been  told  by  a  doctor,  nurse,  or  other  health  professional 

that  you  have  high  blood  pressure?  (45) 

a.  Yes 1 

b.  No     Go  to  Q.  12  (p.  8)   2 

Don't  know/Not  sure     Go  to  Q.  12  (p.  8) 7 

Refused     Go  to  Q.  12  (p.  8)  9 

1 1 .  Have  you  been  told  on  more  than  one  occasion  that  your  blood  pressure 

was  high,  or  have  you  been  told  this  only  once?  (46) 

a.  More  than  once 1 

b.  Only  once 2 

Don't  know/Not  sure 7 

Refused 9 
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Section  4:  Cholesterol  Awareness 

12.  Blood  cholesterol  is  a  fatty  substance  found  in  the  blood.  Have  you  ever 

had  your  blood  cholesterol  checked?  (47) 

a.  Yes 1 

b.  No     Go  to  Q.  15  (p.  9) 2 

Don't  know/Not  sure      Go  to  Q.  15  (p.  9) 7 

Refused      Go  to  Q.  15  (p.  9) 9 

13.  About  how  long  has  it  been  since  you  last  had  your  blood 

cholesterol  checked?  (48) 

Read  Only  if  Necessary 

a.  Within  the  past  year  (1  to  12  months  ago) 1 

b.  Within  the  past  2  years  ( 1  to  2  years  ago) 2 

c.  Within  the  past  5  years  (2  to  5  years  ago) 3 

d.  5  or  more  years  ago  4 

Don't  know/Not  sure 7 

Refused 9 

14.  Have  you  ever  been  told  by  a  doctor  or  other  health  professional 

that  your  blood  cholesterol  is  high?  (49) 

a.  Yes 1 

b.  No 2 

Don't  know/Not  sure 7 

Refused 9 
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5  packs  = 
100  cigarettes 


Section  7:  Tobacco  Use 

21.  Have  you  smoked  at  least  100  cigarettes  in  your  entire  life?  (57) 

a.  Yes 1 

b.  No     Go  to  Q.  27  (p.  15)   2 

Don't  know/Not  sure     Go  to  Q.  27  (p.  15) 7 

Refused     Go  to  Q.  27  (p.  15)  9 

22.  Do  you  smoke  cigarettes  now?  (58) 

a.  Yes 1 

b.  No     Go  to  Q.  26  (p.  14) 2 

Refused     Go  to  Q.  27  (p.  15)  9 

23.  On  how  many  of  the  past  30  days  did  you  smoke  cigarettes?  (59-60) 

a.  Number  of  days    If  less  than  30,  go  to  Q.  24a  (p.  14) 

b.  None     Go  to  Q.  26  (p.  14) 8  8 

Don't  know/Not  sure 7  7 

Refused 9  9 

24.  On  the  average,  about  how  many  cigarettes  a  day  do  you  now  smoke?  (61-62) 

Number  of  cigarettes     Go  to  Q.  25  (p.  14) 

Don't  know/Not  sure     Go  to  Q.  25  (p.  14) 7  7 

Refused     Go  to  Q.  25  (p.  14)  9  9 


1  pack  = 
20  cigarettes 
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24a.  On  the  average,  when  you  smoked  during  the  past  30  days,  about  how 

many  cigarettes  did  you  smoke  a  day?  (63-64) 


1  pack = 
20  cigarettes 


Number  of  cigarettes     Go  to  Q.  27  (p.  15) 

Don't  know/Not  sure     Go  to  Q.  27  (p.  15) 7  7 

Refused     Go  to  Q.  27  (p.  15)  9  9 

25.  During  the  past  12  months,  have  you  quit  smoking  for  1  day  or  longer?  (65) 

a.  Yes     Go  to  Q.  27  (p.  15)    1 

b.  No    Go  to  Q.  27  (p.  15)   2 

Don't  know/Not  sure     Go  to  Q.  27  (p.  15) 7 

Refused     Go  to  Q.  27  (p.  15)  9 

26.  About  how  long  has  it  been  since  you  last  smoked  cigarettes  regularly, 

that  is,  daily?  (66-67) 

Read  Only  if  Necessary 

a.  Within  the  past  month  (0  to  1  month  ago) 0  1 

b.  Within  the  past  3  months  (1  to  3  months  ago) 0  2 

c.  Within  the  past  6  months  (3  to  6  months  ago) 03 

d.  Within  the  past  year  (6  to  12  months  ago) 0  4 

e.  Within  the  past  5  years  (1  to  5  years  ago) 0  5 

f.  Within  the  past  15  years  (5  to  15  years  ago)  0  6 

g.  1 5  or  more  years  ago  0  7 

Don't  know/Not  sure 7  7 

Never  smoked  regularly 8  8 

Refused 9  9 
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Module  2:  Fruits  and  Vegetables 

These  next  questions  are  about  the  foods  you  usually  eat  or  drink.  Please  tell  me 
how  often  you  eat  or  drink  each  one,  for  example,  twice  a  week,  three  times  a 
month,  and  so  forth.  Remember,  I  am  only  interested  in  the  foods  you  eat. 
Include  all  foods  you  eat,  both  at  home  and  away  from  home. 

1 .  How  often  do  you  drink  fruit  juices  such  as  orange,  grapefruit, 

or  tomato?  (148-150) 

a.  Per  day 1 

b.  Per  week 2 

c.  Per  month 3 

d.  Per  year 4 

e.  Never 5    5    5 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 

2.  Not  counting  juice,  how  often  do  you  eat  fruit?  (151-153) 

a.  Per  day 1 

b.  Per  week 2 

c.  Per  month 3 

d.  Per  year 4 

e.  Never 5    5    5 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 
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3.  How  often  do  you  eat  green  salad?  (154-156) 

a.  Per  day 1 

b.  Per  week 2 

c.  Per  month 3 

d.  Per  year 4 

e.  Never 5    5    5 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 

4.  How  often  do  you  eat  potatoes,  not  including  french  fries, 

fried  potatoes,  or  potato  chips?  (157-159) 

a.  Per  day 1 

b.  Per  week 2 

c.  Per  month 3 

d.  Per  year 4 

e.  Never 5    5    5 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 
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5.     How  often  do  you  eat  carrots?  (160-162) 

a.  Per  day 1 

b.  Per  week 2 

c.  Per  month 3 

d.  Per  year 4 

e.  Never 5    5    5 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 


6.     Not  counting  carrots,  potatoes,  or  salad,  how  many  servings  of 

vegetables  do  you  usually  eat?  (163-165) 


Example:  a 

serving  of 

vegetables 

at  both  lunch 

and  dinner 

would  be 

two  servings. 


a.  Per  day 1 

b.  Per  week 2 

c.  Per  month 3 

d.  Per  year 4 

e.  Never 5    5    5 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 
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Module  4:  Exercise 

The  next  few  questions  are  about  exercise,  recreation,  or  physical  activities 
other  than  your  regular  job  duties. 

1 .     During  the  past  month,  did  you  participate  in  any  physical  activities 
or  exercises  such  as  running,  calisthenics,  golf,  gardening,  or  walking 
for  exercise?  (184) 

a.  Yes 1 

b.  No     Go  to  Next  Module 2 

Don't  know/Not  sure     Go  to  Next  Module   7 

Refused     Go  to  Next  Module    9 


2.     What  type  of  physical  activity  or  exercise  did  you  spend  the  most 

time  doing  during  the  past  month?  (185-186) 

Activity  (specify): 


See  coding  list  A 
Refused     Go  to  Q.  6 9    9 


Ask  Q.  3  only  if  answer  to  Q.  2  is  running,  jogging, 
walking,  or  swimming.  All  others  go  to  Q.  4. 


3.     How  far  did  you  usually  walk/run/jog/swim?  (187-189) 

Miles  and  tenths . 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 


See  coding 
list  B  if 
response  is 
not  in  miles 
and  tenths. 


4.      How  many  times  per  week  or  per  month  did  you  take  part  in  this 

activity  during  the  past  month?  (190-192) 

a.  Times  per  week 1 

b.  Times  per  month 2 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 
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5.     And  when  you  took  part  in  this  activity,  for  how  many  minutes 
or  hours  did  you  usually  keep  at  it? 


(193-195) 


Hours  and  minutes 


Don't  know/Not  sure 


111 


Refused 


9    9    9 


6.     Was  there  another  physical  activity  or  exercise  that  you  participated 
in  during  the  last  month? 


a.    Yes 


(196) 
1 


b.    No     Go  to  Next  Module 2 

Don't  know/Not  sure     Go  to  Next  Module 7 


Refused     Go  to  Next  Module 9 


7.     What  other  types  of  physical  activity  gave  you  the  most  exercise 
during  the  past  month? 


(197-198) 


Activity  (specify): 


See  coding  list  A 
Refused     Go  to  Next  Module  


o 


Ask  Q.  8  only  if  answer  to  Q.  7  is  running,  jogging, 
walking,  or  swimming.  All  others  go  to  Q.  9. 


9  9 


8.     How  far  did  you  usually  walk/run/jog/swim? 


(199-201) 


See  coding 
list  B  if 
response  is 
not  in  miles 
and  tenths. 


Miles  and  tenths 


Don't  know/Not  sure 


111 


Refused 9    9    9 
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9.  How  many  times  per  week  or  per  month  did  you  take  part  in  this 

activity?  (202-204) 

a.  Times  per  week 1 

b.  Times  per  month 2 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 

1 0.  And  when  you  took  part  in  this  activity,  for  how  many  minutes 

or  hours  did  you  usually  keep  at  it?  (205-207) 

Hours  and  minutes : 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 
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Activity  Codes  and  Intensity  Factors  for  Common  Leisure  Activities 

Coding  List  A 


Code  description 

01.  Aerobics  class 

02.  Backpacking 

03.  Badminton 

04.  Basketball 

05.  Bicycling  for  pleasure 

06.  Boating  (canoeing,  rowing, 

sailing  for  pleasure  or  camping) 

07.  Bowling 

08.  Boxing 

09.  Calisthenics 

10.  Canoeing/rowing — in  competition 

1 1 .  Carpentry 

12.  Dancing-aerobics/ballet 

13.  Fishing  from  river  bank  or  boat 

14.  Gardening  (spading,  weeding, 

digging,  filling) 

15.  Golf 

16.  Handball 

17.  Health  club  exercise 

18.  Hiking — cross-country 

19.  Home  exercise 

20.  Horseback  riding 

21.  Hunting  large  game — deer,  elk 

22.  Jogging 

23.  Judo/karate 

24.  Mountain  climbing 

25.  Mowing  lawn 

26.  Paddleball 

27.  Painting/papering  house 


Code  description 

28.  Racketball 

29.  Raking  lawn 

30.  Running 

3 1 .  Rope  skipping 

32.  Scuba  diving 

33.  Skating — ice  or  roller 

34.  Sledding,  tobogganing 

35.  Snorkeling 

36.  Snowshoeing 

37.  Snow  shoveling  by  hand 

38.  Snow  blowing 

39.  Snow  skiing 

40.  Soccer 

41.  Softball 

42.  Squash 

43.  Stair  climbing 

44.  Stream  fishing  in  waders 

45.  Surfing 

46.  Swimming  laps 

47.  Table  tennis 

48.  Tennis 

49.  Touch  football 

50.  Volleyball 

51.  Walking 

52.  Waterskiing 

53.  Weight  lifting 

54.  Other 

55.  Bicycling  machine  exercise 

56.  Rowing  machine  exercise 


Coding  List  B 


Lap  Swimming 

Size  pool 

50  ft.  pool 
100  ft.  pool 
50  meter  pool 


Laps 

10  laps  =  .1  mile 
5  laps  =  .  1  mile 
3  laps  =  .  1  mile 


Running/Jogging/Walking 

1/2  mile  =  .5  mile 

1/4  mile  =  .3  mile 

1/8  mile  =  .1  mile 

1  block  =  .  1  mile 
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Module  5:  Weight  Control 

1.  Are  you  now  trying  to  lose  weight?  (208) 

a.  Yes     Go  to  Q.  3 1 

b.  No 2 

Don't  know/Not  sure 7 

Refused 9 

2.  Are  you  now  trying  to  maintain  your  current  weight,  that  is  to  keep 

from  gaining  weight  (209) 

a.  Yes 1 

b.  No     Go  to  Q.  6 2 

Don't  know/Not  sure     Go  to  Q.  6 7 

Refused     Go  to  Q.  6 9 

3.  Are  you  eating  either  fewer  calories  or  less  fat  to... 
lose  weight?/*/ "Yes" on  Q.  1] 

keep  from  gaining  weight?  [if  "Yes"  on  Q.  2]  (210) 

a.  Yes,  fewer  calories 1 

b.  Yes,  less  fat 2 

c.  Yes,  fewer  calories  and  less  fat 3 

d.  No 4 

Don't  know/Not  sure 7 

Refused 9 


Probe 

for 

which 
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4.  Are  you  using  physical  activity  or  exercise  to... 
lose  weight?  [if  "Yes"  on  Q.  1] 

keep  from  gaining  weight?  [if  "Yes"  on  Q.  2]  (211) 

a.  Yes 1 

b.  No 2 

Don't  know/Not  sure 7 

Refused 9 

5.  How  much  would  you  like  to  weigh?  (212-214) 

Weight  

pounds 

Don't  know/Not  sure 7    7    7 

Refused 9    9    9 

6.  In  the  past  12  months,  has  a  doctor,  nurse,  or  other  health  professional 

given  you  advice  about  your  weight?  (215) 

a.  Yes,  lose  weight 1 

b.  Yes,  gain  weight 2 

c.  Yes,  maintain  current  weight 3 

d.  No 4 

Don't  know/Not  sure 7 

Refused 9 


Probe 

for 

which 
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